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ABSTRACT 


As software systems increase in size and complexity, so does the need to 
predict and control scope, schedule, and costs. The United States General 
Accountability Office has acknowledged weaknesses in the software acquisition 
process. Industry data indicates that improving the software development 
process can have significant effect on a project team’s ability to generate 
products within planned scope, schedule, and cost estimates. This thesis focus 
is on software maintenance, one phase of the Army’s acquisition process, to 
demonstrate that stronger management practices are needed to make better 
predictions and assessments in those areas. Two software maintenance 
projects were evaluated for success in project management performance against 
CMMI practices. This research results in a set of recommendations and 
predicted benefits are provided for use by the organization as input to the next 
process improvement effort. 
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I. 


INTRODUCTION 


As software systems increase in size and complexity, the need to predict 
and control costs increases correspondingly. The United States General 
Accountability Office has acknowledged weaknesses in the software acquisition 
process in their March 2004 “Report to the Committee on Armed Services, U. S. 
Senate, Defense Acquisitions, Stronger Management Practices Are Needed to 
Improve DOD’s Software-Intensive Weapon Acquisitions” which indicates that 
"... For the most part, in the programs reviewed, DOD garnered poor results 
from its software acquisition process because it has not employed consistent 
practices in these areas [setting product requirements, maintaining a disciplined 
development process, and using metrics] [6]”. Effective definition and use of 
software metrics in the software development process can assist project 
managers in making decisions and managing risks that affect the extent of 
project scope, schedule, and cost. Collecting scope, schedule, and cost estimate 
data, and then comparing actual results to that data over time can help to answer 
questions about the progress of a project and improve the accuracy of estimates 
for future projects. If the scope, schedule, and budget are not progressing as 
good as planned, software teams and managers can use the information to make 
more informed mid-course decisions to help manage project risks. Changes in 
software development project costs can translate into major program level 
savings or losses, in terms of scope, schedule, cost, and quality. The ability to 
predict the cost of a software project (or the portions of large systems that are 
made up of software) has a large affect on the success of requesting and 
planning the funding needed in such a way that actual costs turn out to be as 
close as possible to expected costs. 

Many DOD organizations use Capability Maturity Models developed by the 
Software Engineering Institute to assess the ability to manage development, 
acquisition, and maintenance (or “health” of a project) of products [7], The value 
of these models has been recognized in typical government contract language, 
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which recommends that bidders have achieved (or have a plan to achieve) a 
minimum process maturity level as one of the bidding requirements. Edward 
Aldridge, Under Secretary of Defense (Acquisition, Technology and Logistics), 
issued a March 2003 “Memorandum for Secretaries of the Military Departments” 
with reference to Section 804 of the Bob Stump National Defense Authorization 
Act for Fiscal Year 2003, which requires establishment of software acquisition 
process improvement programs for defense agencies that manage major 
acquisition programs [5]. In particular, the Army has recognized that software 
engineering is an area that is in need of attention [9], 

As a result of these military directions, software management 
organizations are encouraged to implement software development process 
improvement programs. Aside from improving the organizations’ ability to deliver 
the most and best products, within allotted time and budget, the primary 
motivation is to demonstrate capability for bidding on future contracts [10]. 

This thesis focus is on software maintenance, one phase in the Army’s 
acquisition process. The purpose of the study is to demonstrate that stronger 
management practices are needed. Two software maintenance projects 
(referred to as “Case 1” and “Case 2”) were evaluated for success in these areas 
against CMMI practices [2], To keep the results manageable for this thesis, only 
three CMMI Process Areas were considered. Results of the evaluations were 
examined to reveal answers to the following questions: 

• Was the organization able to manage technical and managerial 
requirements? Were inconsistencies between requirements and 
project plans/products identified and managed? 

• Was the organization able to establish and maintain project baselines? 

• Was the organization able to provide an understanding of project 
progress so that corrective action could be taken to make changes in 
project management baselines? 


2 



• If not, what process improvements would need to be implemented to 
be proficient in these practices? 

• What benefits might the organization realize as a result of this process 
implementation? 

For proprietary reasons, the organization and particular projects studied 
will remain anonymous. While both project teams were able to deliver the 
software within the planned scope, schedule, and cost constraints, more effective 
project management may have helped them to finish sooner, or to take on more 
scope within the same limits. The information collected by this study will be 
available for immediate use by the project team as they are preparing for the next 
software maintenance project, while at the same time working towards achieving 
Level 3 goals of the CMMI. 

Note that the informal examination of project management practices 
against three CMMI Process Areas, demonstrated on two similar software 
maintenance projects conducted for this study, will be referred to as an 
“evaluation.” This term was chosen to distinguish the activity from an “audit” 
which implies that a corrective action loop is included. The goal was simply to 
evaluate the projects, compare actual practices to related CMMI practices, and 
compare them to each other and to the lessons learned reported by the project 
teams. The term “evaluation” is used to distinguish this activity from a process 
improvement “assessment” which implies the involvement of independent 
licensed professionals. Instead, an informal method was developed and used to 
extract information strictly for internal use. However, it is expected that a formal 
SEI CMMI assessment would yield the same results. 

This thesis documents the steps taken to set up and perform the process 
evaluations. As a result, the author compiled a list of software development 
project management process improvement suggestions and potential benefits, 
which includes successful achievement of the goals in each of the three CMMI 
Process Areas considered. 
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A. ASSERTIONS 

The SEI’s process improvement model (the CMMI) is an accepted and 
good method to determine the level of process maturity in a software 
organization. Once conducted, evaluation results can be used to target specific 
areas for improvement. A comparison of the lessons learned reported by project 
team can be used as a sanity check against evaluation results. If the project 
team reported problems that are related to deficient practices then evaluation 
results may be more believable to them, which may help to give recommended 
improvements more credibility. Finally, if process improvement suggestions are 
implemented, benefits are likely (i.e., project managers should be able to rely on 
quantitative measurement as a tool to build greater accuracy and confidence into 
planning and progress reporting). 

B. SCOPE 

This study is limited to process improvement of project management 
activities and excludes discussion of any other areas of software development 
process improvement or software development activities. Two separate, but 
similar, real-world software maintenance projects were evaluated for the purpose 
of developing thesis conclusions. The evaluations consist of a comparison of the 
two projects’ management practices to those defined in three particular CMMI 
Process Areas, which were selected because they describe a standard for 
activities that relate to planning and tracking requirements, schedule, and cost. 
The three Process Areas consist of: 1) Requirements Management, 2) Project 
Planning, and 3) Project Monitoring and Control. The primary intention of the 
research is to produce a relative picture of the two projects, and the differences 
between actual practices and the selected CMMI practices to provide a launch 
point for process improvement efforts internal to the organization. Since 
improvement is a continuous process, further validation and verification of thesis 
conclusions (predicted benefits) will require implementation and analysis of the 
improvements suggested. 
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C. THESIS ORGANIZATION 

Following the INTRODUCTION, BACKGROUND, and METHOD chapters, 
a chapter on the study of each of the three relevant CMMI Process Areas is 
introduced (REQUIREMENTS MANAGEMENT STUDY, PROJECT PLANNING 
STUDY, and PROJECT MONITORING AND CONTROL STUDY). Each of those 
chapters presents the CMMI evaluation results for both projects evaluated, the 
post-mortem lessons learned reported by the project team, an analysis of the 
data, and suggestions for improvement in each area. The POTENTIAL 
BENEFITS chapter is then provided as a basis for process improvement 
motivation for the organization, and perhaps for other Army software 
maintenance organizations, or Army organizations that depend on software 
maintenance organizations for product delivery. It outlines the predicted return 
values that could result from implementation of the suggested process 
improvements. 
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II. 


BACKGROUND 


A summary of some basic concepts, either used by the projects studied, 
or used as a standard to compare project practices to, is provided in this chapter. 
Specifically, a summary of CMMI Levels 1-3 project management goals and 
practices, and project management process requirements in terms of Unified 
Modeling Language notation are presented to set the stage for this research. 
This information was considered as the basis for the evaluations and analysis to 
follow. 

A. PROJECT MANAGEMENT IN THE CONTEXT OF THE CMMI 

The CMMI groups Process Areas for the purpose of discussing concepts 
and interactions. Overall, there are seven Level 2 Process Areas (PAs) and 
fourteen Level 3 PAs, each with a set of goals defined for achievement. 


Maturity Levels 



Figure 1. “CMMI Model Components (From [1])” 


There are three CMMI Process Areas that address activities surrounding 
the estimation, collection, tracking, and reporting of the project management data 
mentioned above. These Level 2 PAs are Requirements Management, Project 
Planning, and Project Monitoring and Control. PAs are divided into Specific and 
Generic Practices, as shown in Figure 1 [1]. Specific Practices are model 
components that are considered important guidance for achieving “specific” 
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goals. Generic goals and practices are required model components that apply to 
all Process Areas. Each project was evaluated in all three PAs, in both specific 
and generic practices. A more detailed description of the three process areas is 
provided in the following subsections. 

1. Basic Project Management Process Areas 

The CMMI states, “the purpose of Project Planning (PP) is to establish 
and maintain plans that define project activities” and “the purpose of Project 
Monitoring and Control (PMC) is to provide an understanding of the project’s 
progress so that appropriate corrective actions can be taken when the project’s 
performance deviates significantly from the plan [2].” Together, these two 
Process Areas are part of the set associated with fundamental project 
management activities and thus were selected for this research. CMMI presents 
the whole group of project management PAs as shown in Figure 2 [4], 



Figure 2. “Basic Project Management Process 
Areas (From [4])” 
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2. Engineering Process Areas 

The CMMI states, “The purpose of Requirements Management (REQM) is 
to manage the requirements of the project’s products and product components, 
and to identify inconsistencies between those requirements and the project’s 
plans and work products [4].” In the case of software maintenance performed by 
the organization, a requirement refers to the “Statement of Work” which is 
approved by the customer. It includes project requirements (schedule and cost 
constraints) and technical requirements (a statement of work). This Process 
Area is part of a group defined by the CMMI as the “Engineering Process Areas.” 
Figure 3 [4] shows that requirements management is an activity that affects the 
engineering process and continues with a loop of feedback and control 
throughout the project. 



Figure 3. “Engineering Process Areas (From [4])” 
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B. PROJECT MANAGEMENT IN THE CONTEXT OF UML 


Another way to view process requirements is to use Unified Modeling 
Language, which provides a visual representation of a system (in this case, a 
project management system) from the users (users of the process) perspective. 
Abstraction can be used to define concepts and relationships between system 
components without cumbersome detail. 

1. Project Management System Functions 

At the most general level, a Project Management System can be shown 
as a collection of elements in Figure 4. The box shows the boundary of the 
system, the class diagrams show the subsystems whose functionalities 
(functional requirements of the process) can be expressed as use cases in the 

following subsections. _ 

Project Management System 



Figure 4 Project Management System 
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Further detailed notation of each subsystem includes actors that represent 
points of interaction. Connections between these symbols describe the 
interaction needed to accomplish the objective of the use case (subsystem 
behavior). The following subsections describe each element of the project 
management system. These diagrams provide visual representation of the 
benchmark to be used as a comparison to the process performance of the 
projects studied. Each element in the subsystem use case diagrams addresses 
the requirements and interactions that represent the behaviors described in the 
CMMI practices for that Process Area, now with the participants (or, actors) 
included. 

2. Requirements Management Element 

Requirements management occurs both external and internal to the 
organization. At this level of abstraction, these aspects are not distinguishable. 
What can be seen is that for this subsystem, three actors are involved in the five 
practices shown in Figure 5. 
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Figure 5. Requirements Management Use Case Diagram 

3. Project Planning Element 

Project Planning includes activities both internal and external to the 
organization. At this level of abstraction these aspects are not distinguishable. 
What can be seen is that for this subsystem, two actors are involved in the eight 
practices shown in Figure 6. 
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Figure 6. Project Planning Use Case Diagram 

4. Project Monitoring and Control Element 

Project Monitoring and Control includes activities both internal and 
external to the organization. At this level of abstraction these aspects are not 
distinguishable. What can be seen is that for this subsystem, two actors are 
involved in the eight practices shown in Figure 7. 


13 





Figure 7. Project Monitoring and Control Use Case Diagram 
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III. METHOD 


Two projects were selected for this study. The projects had similar project 
teams, requirements, schedules, artifacts, and staff. A checklist was developed 
for each of the three CMMI Process Areas chosen for the study (see sample in 
Appendix). A rating system to categorize the level of performance of each 
practice was developed, utilizing one of three possible outcomes for each 
practice evaluated: “Not Performed,” “Not Performed Adequately,” “Performed 
Adequately (or Better).” It is understood that two of the performance categories 
are subjective, and were designed to reveal the amount of improvement that 
would be necessary in certain areas. Results from both Case 1 and Case 2 were 
recorded on the same evaluation checklist. Information included in the evaluation 
results was provided to and reviewed by the project team members who 
participated in the evaluation. 

A. PROCESS EVALUATION 

1. Entry Criteria 

The conditions (work products, activities, or events) that must have taken 
place before the activities described in the Procedure can be executed are as 
follows: 

• Selection of the CMMI as the model as a benchmark for comparison 
purposes 

• Development of checklists, rating scale, approach, and goals of 
process evaluations 

• Ensure that the project team participants understand the criteria (CMMI 
practices) for comparison, the goals, and the rating system to be used 

• Obtain permission from the organization’s senior management for 
team members to participate in reviewing and discussing evaluation 
results 
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2. Input 

Information, work products used by this procedure, or actions that are 
needed before the procedure can be executed are as follows: 

• Previous project lessons learned (The lessons were derived from 
comments provided by project team members on an initial anonymous 
survey and from subsequent feedback sessions held with the project 
team to discuss the survey’s findings.) 

• Project artifacts such as project plans, project/senior management 
review/customer meeting minutes, configuration and quality assurance 
records, etc. 

• CMMI Process Area Evaluation Checklists (see Appendix for example) 

3. Exit Criteria 

The list of the conditions (completed work products or actions) that are 
needed before the procedure can be considered complete and the next activity 
can begin is as follows: 

• Collection of information such that a basis for a rating on each practice 
can be formed 

• Consensus by participants, or resolution of disputes by the SEPG 
Lead, on evaluation results 

4. Output 

The list of the benefits and work products produced or actions that will 
result when the three evaluations are complete is as follows: 

• Better understanding of the processes used by the organization 

• Better understanding of the CMMI 

• Evaluation results for two software maintenance projects in terms of 
suggested CMMI practices for three chosen Process Areas 

• Better understanding of the organization’s current capability 

• Better understanding of the gaps between the organization’s practices 
and suggested CMMI practices 
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• Better understanding of how the organization could do process audits 

• Process evaluation checklists that can be tailored later to add more 
organization specific processes, and then used as process audit 
checklists for internal CMMI assessments 

• Outlined procedures were generated for some procedures that were 
not documented prior to the evaluation 

• Process Improvement artifacts (evaluation results demonstrate 
whether the organization has improved, maintained, or digressed in 
certain areas) from project to project 

• Draft procedure for conducting process audits 

• A sense of accomplishment, as a team, for having reached consensus 
together. Project team members worked together with SEPG 
members (some wore both hats). 

5. Procedure 

The following list summarizes the steps taken to conduct the process 
evaluations: 

• Evaluation team participants were briefed on the approach, goals, and 
review checklists by the evaluator (author of this paper). A consensus 
on format, content, and rating system of the checklists was obtained. 
Checklists were revised by the evaluator as recommended. 

• The process evaluator reviewed project artifacts, and when necessary, 
conducted interviews with project members to get input on processes 
and artifacts used. 

• The process evaluator drafted evaluation results based on research 
and initial feedback. 

• Draft evaluation results were released to the evaluation team for 
review. 
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• The evaluator and evaluation team met to discuss results. The rating 
and justification for each rating were reviewed for each practice, and 
each project. The evaluator noted clarification and identification of 
missing artifacts or supporting information. 

• The evaluator finalized evaluation results by making recommended 
changes. 

• The evaluator released the results to the evaluation team for final 
review, and to a few additional project team members who may have 
been able to offer additional feedback. Once all feedback was 
incorporated, the results were considered final. 

• The evaluator and evaluation team presented the evaluation results to 
the project team together. 

B. ANALYSIS 

The next three chapters present discussion from a variety of perspectives 
on observed project management ability. First, a summary of both Case 1 and 
Case 2 CMMI Process Area evaluation results is presented. Analysis of each 
project is then presented. In addition to presentation of a brief summary of 
evaluation results, each analysis section includes review of post mortem lessons 
learned (which were reported by project team members). Next, a discussion on 
the comparison of Case 1 to Case 2 evaluation results shows whether or not the 
organization has improved, maintained, or digressed in process capability given 
the goals and practices examined. A comparison of the lessons learned to the 
evaluation results was made to determine if there is a correlation between the 
comments and the gap in practices. The purpose is to determine if there are 
similarities between the lessons learned and the evaluation results. 

Rating categories can be viewed as a measure of the amount of 
improvement that will be required for the organization to reach success in each 
practice (i.e., “Not Performed” means there is room for every improvement, “Not 
Performed Adequately” means there is still room for significant improvement, and 
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“Performed Adequately” means that there is still room for some improvement). 
Process improvement is never done! 
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IV. REQUIREMENTS MANAGEMENT STUDY 


The results and analysis of the requirements management process for the 
two case studied are shown in the following subsections. An outline of the goals 
and associated practices defined for the CMMI Requirements Management 
Process Area, a score for each project studied (Case 1 and Case 2), and 
justification for each rating is first provided. A general analysis of the evaluation 
results for each case follows the data presentation. The analysis includes 
presentation and comparison to the post-mortem lessons learned to determine if 
there are correlations with the evaluation results. Note that the lessons are 
unique to the individual projects, and that only the lessons that apply to the area 
of study are included. Finally, a comparison of the two cases is made and a list 
of suggested improvements is provided. 

A. CASE 1 AND CASE 2 EVALUATION RESULTS 

CMMI interpretation notes, the score, and the basis for the rating for the 
Case 1 and Case 2 projects are provided below: 

• Specific Goal (SG) 1 - “Requirements are managed and 

inconsistencies with project plans and work products are identified.” 
o Specific Practice (SP) 1.1 - “Develop an understanding with the 
requirements providers on the meaning of the requirements” was 
“Performed Adequately’ for both Case 1 and Case 2. 

1. “Requirements” was interpreted as the statement of work, 
budget, and delivery schedule constraints. 

2. The organization demonstrated that they worked externally 
with the industry partner and customer to get more familiar 
with the requirements for both projects. The allocation of 
requirements to the organization and industry partner was 
then negotiated, and an agreement was made with the 
customer. 
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3. Internally, development and test personnel were involved in 
validating software trouble reports before they were added to 
the scope, thereby demonstrating their understanding of the 
project scope. 

o SP1.2 - “Obtain Commitment to the requirements from the project 
participants” was “Performed Adequately” for Case 1. The 
practice was “Not Performed Adequately’ for Case 2. 

1. Note that this practice was interpreted as referring to “initial” 
commitment made to a specific set of requirements. 

2. The organization demonstrated that an external initial 
commitment was made as a result of negotiation of the 
project requirements with the industry partner, and then 
approved by the customer. 

3. The organization demonstrated internal commitment by 
involving the organization project management, functional 
leads (software development, test, CM, and SQA) in the 
planning activities, including peer review of the project 
plan(s). A baseline is created for this organization following 
successful senior management review and approval. 

4. The Case 1 project team was successful at completing the 
above process. The Case 2 project team did not get the 
project plan through the peer review process until very late in 
the program, and did not get senior management approval 
until after the work was done, thus received the lesser rating. 
As a result, during the program the test and development 
teams were well aware of the schedule. However, the CM 
and SQA groups were not fully involved in these planning 
activities and were forced to accommodate the schedule 
after the fact. 
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o SP1.3 - “Manage changes to the requirements as they evolve 
during the project” was “Not Performed Adequately’ for both Case 
1 and Case 2. 

1. This practice was interpreted as referring to changes made 
after initial baselines were established. 

2. At the program level, as requirements changed for any 
reason, the baseline statement of work was reviewed, 
updated, and approved. 

3. As a result, the internal schedule and scope were updated, 
informally for both Cases. Peer reviews to make sure all 
functional teams were well aware of the changes, and the 
senior management approval process were neglected for the 
Case 1 baseline. The Case 2 baseline was never 
established to begin with. 

o SP1.4 - “Maintain bi-directional traceability among the 

requirements and the project plans and the work products” was 
“Not Performed Adequately’ for both Case 1 and Case 2. 

1. This practice was interpreted as referring to both the initial 
set of requirements, and to any changes made to the 
requirements. 

2. Final code and test procedure peer review packages from 
both projects demonstrate that affected requirements were 
accurately identified, and indeed, if requirements changes 
were necessary, system requirements were affected 
accordingly. 

3. However, for both projects, traceability of the requirements 
as stated in the internal project plans were not maintained 
via the planned peer review or senior management approval 
process, thus, the lesser rating was assigned. 
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o SP1.5 - “Identify inconsistencies between the (project plans and 
work products) and the requirements” was “Performed 
Adequately’ for Case 1. The practice was “Not Performed 
Adequately’ for Case 2. 

1. Note parenthesis added for interpretation to the practice 
definition. 

2. As requirements changed, the internal schedule was 
revised, albeit informally. As some requirements were 
eliminated, others were added. Development and test plans 
were revised accordingly. Changes were maintained 
informally by the software development lead. However, the 
changes were not added to the project plans to undergo the 
planned peer review and senior management approval 
process. 

3. Both projects were successful at micro-managing the 
requirements (work and services were performed by 
development, test, CM and SQA groups). However, the 
macro-management (coordination of activities) was missing 
for the Case 2 project. In fact, there was a major schedule 
misunderstanding between the organization and the industry 
partner. This could have been due to the lack of a dedicated 
project manager. The project team was able to recover 
anyway. 

4. Note that because of the small size of the project ream 
(around 20 members), a high-level of daily face-to-face 
communication took place to help maintain smooth 
interfaces. In addition, many of the same project members 
from the Case 2 team worked on the Case 1 project, and on 
a similar project prior to the work done for Case 1. The team 
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was able to rely on established relationships and processes 
to help with the success of the project. 

• Generic Goal (GG) 2 - “The process is institutionalized as a managed 
process.” 

o Generic Practice (GP) 2.1 - “Establish and maintain an 

organizational policy for planning and performing the Requirements 
Management process” was “Performed Adequately’ for both Case 
1 and Case 2. 

1. The organization does have a formulated and documented 
Requirements Management policy that is used by the project 
teams. The plans for each of the projects provided a 
statement declaring that no special process tailoring was 
needed to execute the plan. 

o GP2.2 - “Establish and maintain the plan for performing the 
Requirements Management process” was “Performed 
Adequately ’ for both Case 1 and Case 2. 

1. Both project plans (which included the plans for the project’s 
requirements management process) were developed, 
reviewed, and agreed to by team members. 

2. The project team decided that it was immaterial that the 
Case 2 project plan did not complete the peer review or 
senior management approval process successfully. 

o GP2.3 - “Provide adequate resources for performing the 
Requirements Management process, developing the work products, 
and providing the services of the process” was “Performed 
Adequately ’ for both Case 1 and Case 2. 

1. Resources for conducting the projects requirements 
management process were assigned by the project plans, 
o GP2.4 - “Assign responsibility and authority for performing the 
process, developing the work products, and providing the services 
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of the Requirements Management process” was “Performed 
Adequately’ for both Case 1 and Case 2. 

1. Each project plan had a “roles and responsibility” section, 
which assigned the responsibilities and authority to all of the 
requirements management activities. 

o GP2.5 - “Train the people performing or supporting the 
Requirements Management process as needed” was “Performed 
Adequately' for both Case 1 and Case 2. 

1. Training for the project team consisted of previous 
experience, individual education, and on the job training for 
more current experiences. The project team had the skills 
necessary to execute the requirements management 
activities. 

2. For Case 2, the project team did receive training on changes 
to the development and test environment alongside the 
industry partner. 

o GP2.6 - “Place designated work products of the Requirements 
Management process under appropriate levels of configuration” 
was “Not Performed Adequately’ for both Case 1 and Case 2. 

1. The internal project plans (including scope, and schedule 
documents) and other work products were stored in a shared 
computer folder for easy access. Many copies have not yet 
been placed into the “Rational ClearCase” database, which 
is the organizations designated CM tool, thus the lesser 
rating was assigned. 

o GP 2.7 - “Identify and involve the relevant stakeholders of the 
Requirements Management process as planned” was “Performed 
Adequately’ for both Case 1 and Case 2. 

1. Relevant stakeholders (external and internal) were all 
identified in the project plans. While the plans were not 
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formally maintained via the peer review and senior 
management approval processes, all were involved in the 
discussions and decisions made regarding which 
requirements would be added, changed, or deleted. It would 
have been better if the formal processes had been followed, 
as it could have prevented a schedule misunderstanding 
with the industry partner that occurred. The project team 
was able to recover anyway. 

o GP2.8 - “Monitor and control the Requirements Management 
process against the plan for performing the process and take 
appropriate corrective action” was “Performed Adequately' for 
Case 1, and was “Not Performed Adequately’ for Case 2. 

1. As requirements changed, the internal schedule was 
revised, albeit informally. As some requirements were 
eliminated, others were added. Development and test plans 
were revised accordingly. Changes were maintained 
informally by the software development lead. However, the 
changes were not added to the project plans to undergo the 
planned peer review and senior management approval 
process. 

2. Both projects were successful at micro-managing the 
requirements (work and services were performed by 
development, test, CM and SQA groups). However, the 
macro-management (coordination of activities) was missing 
for the Case 2 project. In fact, there was a major schedule 
misunderstanding between the organization and the industry 
partner. This could have been due to the lack of a dedicated 
project manager. The project team was able to recover 
anyway. 
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3. Note that because of the small size of the project team 
(around 20 members), a high-level of daily face-to-face 
communication took place to help maintain smooth 
interfaces. In addition, many of the same project members 
from the Case 2 team worked on the Case 1 project, and on 
a similar project prior to the work done for Case 1. The team 
was able to rely on established relationships and processes 
to help with the success of the project, 
o GP2.9 - “Objectively evaluate adherence of the Requirements 
Management process against its process description, standards, 
and procedures, and address noncompliance” was “Not Performed 
Adequately’ for both Case 1 and Case 2 projects. 

1. The overarching process for SQA discussed independently 
evaluating the requirements management process. This 
was done by SQA at project meetings and senior 
management reviews. However, since a formal audit was 
never conducted the lesser rating was assigned for both 
projects. The project team indicated that a formal audit 
could help to improve the process. 

o GP2.10 - “Review the activities, status, and results of the 
Requirements Management process with higher level management 
and resolve issues” was “Performed Adequately’ for Case 1, and 
was “Not Performed Adequately’ for Case 2. 

1. Requirements development and changes were discussed at 
project and senior management meetings. While both 
meetings were held diligently for the Case 1 project, the 
Case 2 project team conducted only two senior management 
reviews early on in the project. In addition, the occurrence of 
Case 2 project meetings also dropped off dramatically which 
prevented this practice, and thus the lesser rating was 
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assigned. This could have been due to the lack of a 
dedicated project manager. 

• Generic Goal (GG) 3 - “The process is institutionalized a defined 
process.” 

o GP 3.1 - “Establish and maintain the description of a defined 
Requirements Management process” was “Performed 
Adequately’ for both Case 1 and Case 2 projects. 

1. The organization level requirements management procedure 
is a general process description that was used by the project 
teams, and has not changed. Both project plans provided a 
declaration statement indicating that no special process 
tailoring was needed to execute the project activities, 
o GP 3.2 - “Collect work products, measures, measurement results, 
and improvement information derived from planning and performing 
the Requirements Management process to support the future use 
and improvement of the organization’s processes and process 
assets” was “Not Performed Adequately’ for both Case 1 and 
Case 2 projects. 

1. For both projects, the software lead maintained a list of the 
number of initial requirements, how many were deemed not 
applicable, and how many were added. In addition, data on 
the estimates of difficulty and time to resolve problems were 
made, and compared to actual time spent. While this data 
was recorded it was not compiled for analysis with process 
improvement in mind, thus, the lesser rating was assigned. 

B. CASE 1 ANALYSIS 

1. Evaluation Results 

The following observations were made based on process evaluation 
ratings given: 
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• 100% of the practices were performed to some degree (all were rated 
either “Not Performed Adequately’, or “Performed Adequately’ - 
none were rated “Not Performed).” In terms of the CMMI Model, all 
goals in this Process Area have been achieved since all practices are 
being performed to some extent. 

• Overall, 12/17 or 71% of the practices were rated “Performed 
Adequately." This describes the success of most practices associated 
with each of the three goals in this Process Area indicating that the 
improvements will not need to be concentrated on one particular goal, 
rather, a few practices in each area will need to be improved. 

2. Lessons Learned Reported 

The lessons learned that affect the requirements management process is 
described below. 

a. Project Requirements Were Not Well Understood 

Some confusion resulted from the team not having a good 
understanding of project plans (including project goals, schedules, constraints, 
documentation required, processes to be used, the “big picture” of how the 
project management interacts with the industry partner and customer, and any 
organization goals or initiatives to improve project performance). In addition, the 
project team did not know exactly where to look for project data on the share 
drive to obtain up-to-date information on status. 
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C. CASE 2 ANALYSIS 

1. Evaluation Results 

The following observations were made based on process evaluation 
ratings given: 

• 100% of the practices were performed to some degree (all were rated 
either “Not Performed Adequately”, or “Performed Adequately” - none 
were rated “Not Performed).” All practices are being performed to 
some extent. 

• Overall, 8/17 or 47% of the practices were rated “Performed 
Adequately.” This describes the success of most practices associated 
with each of the three goals in this Process Area indicating that the 
improvements will not need to be concentrated on one particular goal, 
rather, a few practices in each area will need to be improved. 

2. Lessons Learned Reported 

a. It Was Hard to Get Next SOW Accepted by Industry Partner 

The SOW for the organization was not finalized and approved 

before work began. This resulted in the industry partner starting/completing work 
on issues that the organization had planned to work on, and in fact, had started 
working on. 

b. There Was Not Enough Coordination of System Integration 
Lab Activities with Developers and Testers 

There was a lack of lab resources when needed (i.e., lab was going 
down so machines were not available for testing). Testing had to be done at the 
industry partner’s lab at the mercy of their schedule. 

D. COMPARISON OF CASE 1 TO CASE 2 

A comparison was done to determine the similarities and differences 
between Case 1 and Case 2 evaluation results. 

1. Goals 

The goals associated with the Requirements Management Process Area 
are as follows: 
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• Specific Goal (SG) 1 - Requirements are managed and inconsistencies 
with project plans and work products are identified. 

• Generic Goal (GG) 2 - The process is institutionalized as a managed 
process. 

• Generic Goal (GG) 3 - The process is institutionalized as a defined 
process. 

The achievement by Case 1 and Case 2 in terms of goal success can be 

described by a comparison of the number of practices given the “Performed 

Adequately” rating to the total number of practices for that goal. The results of 

this analysis follow: 

Case 1 - SG 1 - 3/5 or 60% 

GG 2-8/10 or 80% 

GG 3- 1/2 or 50% 

Case 2- SGI-1/5 or 20% 

GG 2-6/10 or 60% 

GG 3- 1/2 or 50% 

No overall goal improvements can be demonstrated by this data, only 
digression for SG 1 and GG 2. 


2. Practices 


Practice 

Case 1 

Case 2 

Not 

Performed 

Adequately 

Performed 

Adequatel 

y 

Not 

Performed 

Adequately 

Performed 

Adequately 

SP 1.1 


X 


X 

SP 1.2 


X 

X 


SP 1.3 

X 


X 


SP 1.4 

X 


X 


SP 1.5 


X 

X 


GP 2.1 


X 


X 

GP 2.2 


X 


X 

GP2.3 


X 


X 

GP 2.4 


X 


X 

GP 2.5 


X 


X 

GP 2.6 

X 


X 


GP 2.7 


X 


X 

GP 2.8 


X 

X 


GP 2.9 

X 


X 


GP 2.10 


X 

X 
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GP 3.1 


X 


X 

GP 3.2 

X 


X 



Table 1 Summary of Requirements Management Evaluation Results 

The results do not demonstrate improvement from Case 1 to Case 2 in 
any requirements management practice evaluated. They do show digression in 
the following practices (highlighted rows): 

• SP1.2 Obtain Commitment to the requirements from the project 
participants 

• SP1.5 Identify inconsistencies between the project plans and work 
products and the requirements 

• GP2.8 - Monitor and control the Requirements Management process 
against the plan for performing the process and take appropriate 
corrective action 

• GP2.10 - Review the activities, status, and results of the Requirements 
Management process with higher-level management and resolve 
issues. 

In addition, results indicate that for both projects, the following practices 
were “Not Performed Adequately " 

• SP1.3 - Manage changes to the requirements as they evolve during 
the project 

• SP1.4 - Maintain bi-directional traceability among the requirements and 
the project plans and the work products 

• GP2.6 - Place designated work products of the Requirements 
Management process under appropriate levels of configuration 

• GP2.9 - Objectively evaluate adherence of the Requirements 
Management process against its process description, standards, and 
procedures, and address noncompliance 

GP 3.2 Collect work products, measures, measurement results, and 
improvement information derived from planning and performing the 
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Requirements Management process to support the future use and improvement 
of the organization’s processes and process assets. 

3. Comparison of Evaluation Results to Lessons Learned 

Examination of the Requirements Management process evaluation results 
shows that both projects had some practices that were assigned the “Not 
Performed Adequately” rating. Looking closer at the comments reveal that both 
lacked the use of the peer review, senior management approval, and formal CM 
process on project requirements, which resulted in a lack of maintaining 
traceability of project requirements to project plans. 

It makes sense that if project requirement information is not up to date or 
missing in plans, and not accessible, the project team reported, “project 
requirements were not well understood” (the lesson learned for Case 1). 

The Case 2 project had additional practices that received the “Not 
Performed Adequately” rating. Further examination of detailed comments for 
those evaluation results shows that the basic problems included the fact that the 
project team did not get the project plan through the peer review process until 
very late in the program, and did not get senior management approval until after 
the work was done. In addition, macro-management (coordination of activities) 
was missing for the Case 2 project. In fact, there was a major schedule 
misunderstanding between the organization and the industry partner. In addition, 
the Case 2 project team conducted only two senior management reviews early 
on in the project, and the occurrence of project meetings dropped off 
dramatically. The team indicated that this could have been due to the lack of a 
dedicated project manager and Senior Management Reviews were delayed while 
waiting for plan completion. 

One of the lessons learned for Case 2 involved lack of coordination 
between lab activities with developers and testers. The same comment can be 
made (as above), that is, if project requirement information is not up to date or 
missing in plans it was not accessible. As a result, the lack of coordination 

between the project team functional groups is not surprising. For Case 2, this 
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could have been due to the lack of a dedicated project manager. Note that 
because of the small size of the project team (around 20 members) a high-level 
of daily face-to-face communication took place to help maintain smooth 
interfaces. In addition, many of the same project members from the Case 2 team 
worked on the Case 1 project (and on a similar project prior to the work done for 
Case 1). The team was able to rely on established relationships and processes 
to help with the success of the project. 

The evaluation results also showed that while “[process improvement] 
data [this time referring to technical requirements] was recorded, it was not 
compiled for analysis with process improvement in mind,” and that a formal 
process audit [addressing handling of project and technical requirements] had 
not been conducted. 

The other Case 2 lesson learned (It was hard to get the next SOW 
accepted by Industry Partner) resulted from the fact that the SOW for the 
organization was not finalized and approved before work began. This resulted in 
the industry partner starting/completing work on issues that the organization had 
planned to work on, and in fact, had started working on (duplication of effort). 
While this lesson involves technical requirements instead of project 
requirements, the lesson is similar in that it is a logical result if requirement 
information is not up to date or missing in (internal, and/or external) plans. 

It is interesting to note that one characteristic of a CMMI Level 1 
organization is that people create their own processes to do the work, creating 
the potential for duplication, and even conflict later on in the project. Looking 
only at the instance of duplication (the organization and industry partner working 
on the same technical requirements) it might appear that this is a Level 1 
organization. However, Level 2 organizations share lessons learned and best 
practices across projects, or across the organization. Requirements 
Management is a Level 2 Process Area. While the organization did not have 
outstanding performance on all practices, it was demonstrated that something is 

being done by the project team in every area. The transition from simply 
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managing requirements to actually managing plans takes place as the 
organization advances from Level 2 to Level 3. It appears that the source of risk 
lies within managing the plans for this organization. 

This information can be used to identify a business goal on which to base 
targeted process improvement, such as “to improve the project requirements 
baseline and change process.” 

E. SUGGESTED IMPROVEMENTS 

As a result of the above Case 1 and Case 2 analysis, the following list of 
improvement suggestions was compiled. Some suggestions are a direct 
reflection of the lessons learned reported, some are specific to CMMI practice 
requirements that need to be fulfilled, and some are more general and would 
contribute indirectly to all reported deficiencies. The list can be used to improve 
the project performance by targeting process improvement in these areas. 

• Make sure that a project manager is assigned and dedicated to 
initiating and following through on requirements management activities 
for the project. 

• Hold a project kick-off meeting with the project staff to brief the team 
on the project plan(s). Understanding the project requirements will 
help to alleviate some misunderstanding within the project team and 
focus on successfully achieving them in a more efficient manner. 

• Make sure that project plans (including schedule and scope 
requirements and constraints) are reviewed and approved by 
functional leads and senior management early on in the project 
(complete the peer review and management approval process for 
managerial and technical requirements). 

• There needs to be more coordination between the organization’s lab 
and project work. A work plan where all activities are scheduled and 
coordinated with all affected parties should be developed. If 
communication within the project team fails and there isn’t a project 
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manager to keep things on track, senior management must step in to 
avoid negative consequences. 

• The organization needs to get involved with the industry partner earlier 
in the planning stages. In fact, the organization should brief the 
industry partner on the goals of the partnership. 

• Develop questions and metrics to measure accomplishments in 
response to the business goal to “improve the project requirements 
baseline and change process.” 

• Make sure project plans (including schedule and scope documents) 
are updated to reflect program level changes. Use “baselining” 
(accomplished at the organization by completing the senior 
management approval process) to keep a record of current 
agreements made for the duration of the project. 

• Make sure that project and senior management review meetings are 
held as planned, on a regular basis. 

• Place all requirements management work products under formal CM 
control. 

• Conduct a formal audit on the requirements management process at 
least once during project execution. 

• Update policy, process, and procedure documentation to reflect the 
additional activities recommended. 
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V. 


PROJECT PLANNING STUDY 


The results and analysis of the project planning process for the two case 
studied are shown in the following subsections. An outline of the goals and 
associated practices defined for the CMMI Project Planning Process Area, a 
score for each project studied (Case 1 and Case 2), and justification for each 
rating is first provided. A general analysis of the evaluation results for each case 
follows the data presentation. The analysis includes presentation and 
comparison to the post-mortem lessons learned to determine if there are 
correlations with the evaluation results. Note that the lessons are unique to the 
individual projects, and that only the lessons that apply to the area of study are 
included. Finally, a comparison of the two cases is made and a list of suggested 
improvements is provided. 

A. CASE 1 AND CASE 2 EVALUATION RESULTS 

The goals and associated practices defined for the CMMI Project Planning 
Process Area, and individual practice ratings assigned by the project team are as 
follows: 

• Specific Goal (SG) 1 - “Estimates of project planning parameters are 
established and maintained.” 

o Specific Practice (SP) 1.1 - “Establish a top-level work breakdown 
structure (WBS) to estimate the scope of the Project” was 
“Performed Adequately’ for both Case 1 and Case 2. 

1. “Established” was interpreted as formulated, documented, 
and used, and not necessarily baselined. 

2. The initial Work Breakdown Structure for both projects was 
identified in the project schedule. 

3. Case 1 schedule underwent and passed the peer review and 
senior management approval process. 

4. Case 2 schedule did not undergo the peer review or senior 
management approval process. However, the project team 
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indicated that the project team was well aware and working 
to the proposed WBS. The WBS was presented at the 
senior management review meetings, two of which were 
held early on in the project. 

o SP 1.2 - “Establish and maintain estimates of the attributes of the 
work products and tasks” was “Performed Adequately’ for both 
Case 1 and Case 2. 

1. “Attributes” was interpreted as “inputs to estimating models, 
such as complexity, or skills/effort, or estimates for similar 
previous experience (values that affect calculation of the size 
of the project).” 

2. For both Case 1 and Case 2, attributes identified were 
collected and maintained by the functional leads (software 
development, test, CM, and QA) on an informal basis. This 
information was shared between the leads, and some had 
been combined in the project plan. 

o SP1.3 - “Define the project life-cycle phases upon which to scope 
the planning effort” was “Performed Adequately’ for both Case 1 
and Case 2. 

1. For both cases, project life cycle phases were described in 
the project plans as task categories under which effort would 
be organized for recording actual time spent. Estimates 
were separated, and actual effort recorded, under these 
categories. 

o SP1.4 - “Estimate the project effort and cost for the work products 
and tasks based on estimation rationale” was “Performed 
Adequately’ for both Case 1 and Case 2. 

1. For both projects, cost and effort estimates were provided in 
the project plans, and schedule estimates in the project 
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schedules. As noted in the project plans, historical data was 
used to generate the estimates. 

2. For Case 2, the QA Lead documented the comparison and 
analysis of Case 1 estimated and actual effort data to 
generate estimates for Case 2. Other functional leads called 
upon previous experience, instead of examining actual 
values collected, to develop their estimates. 

• Specific Goal (SG) 2 - “A project plan is established and maintained as 
the basis for managing the project.” 

o Specific Practice (SP) 2.1 - “Establish and maintain the project’s 
budget and schedule” was “Not Performed Adequately ’ for Case 
1 or Case 2. 

1. For both projects, cost and effort estimates were provided in 
the project plans, schedule estimates in the project 
schedules. 

2. For Case 1, estimates underwent the peer review and senior 
management approval process. Flowever, estimates were 
not revised when there was a significant scope increase 
early in the project, nor for the remainder of the project. 

3. For Case 2, the estimates did not go through the peer review 
or senior management approval processes, therefore, were 
never baselined (nor maintained). 

o SP2.2 - “Identify and analyze project risks” was “Performed 
Adequately” for Case 1 and “Not Performed Adequately” Case 2. 

1. For both projects, initial risks were identified in the project 
plans, which for Case 1 underwent the peer review and 
senior management approval process (Case 2 risks did not 
go through these processes). 

2. For both projects initial risks were reviewed and updated 
regularly at project and senior management review meetings 
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held early in the projects. A “risk exposure” was calculated 
to sum up the risks identified for each project. 

3. Subsequent review and revision of this data took place at 
regular project and senior management review meetings. 
However, this activity dropped off early in the project for 
Case 2 as the leadership in the project engineer (manager) 
role also dropped off. 

o SP2.3 - “Plan for the management of project data” was 
“Performed Adequately’ for both Case 1 and Case 2. 

1. Plans for managing project data were documented in the 
project plans for each project. 

o SP2.4 - “Plan for the necessary resources to perform the project” 
was “Performed Adequately’ or both Case 1 and Case 2. 

1. Plans for resource allocation were documented in the project 
plans for each project. 

o SP2.5 - “Plan for knowledge and skills needed to perform the 
project” was “Performed Adequately' for both Case 1 and Case 2. 

1. Plans for training needed to accomplish the work were 
documented in the project plans for each project. 

o SP2.6 - “Plan the involvement of identified stakeholders” was 
“Performed Adequately’ for both Case 1 and Case 2. 

1. Identification of (and plans for interaction with) major 
stakeholders was documented in the project plans for each 
project. 

o SP2.7 - “Establish and maintain the overall project plan content” 
was “Not Performed Adequately’ for both Case 1 and Case 2. 

1. For Case 1, the project plan was developed, and passed the 
peer review and senior management approval processes. 
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2. For Case 2, the plan was developed. However, it did not go 
through the same review and approval processes, so an 
initial baseline was never established. 

3. While for both Case 1 and Case 2, project plans were 
established, they were not maintained for either project (to 
reflect changes in scope, schedule, or effort estimates). 

• Specific Goal (SG) 3 - “Commitments to the project plan are 

established and maintained." 

o Specific Practice (SP) 3.1 - “Review all plans that affect the project 
to understand project commitments” was “Performed Adequately’ 
for both Case 1 and Case 2. 

1. For Case 1, the project plan was developed, and passed the 
peer review and senior management approval processes. 

2. For Case 2, the plan was developed, and underwent the 
peer review process. 

o SP3.2 - “Reconcile the project plan to reflect available and 
estimated resources” was “Performed Adequately’ for both Case 
1 and Case 2. 

1. For Case 1, the project plan was developed, and passed the 
peer review and senior management approval processes. 

2. For Case 2, the plan was developed, and underwent the 
peer review process. 

o SP3.3 - “Obtain [initial] commitment from relevant stakeholders 
responsible for performing and supporting plan execution” was 

“Performed Adequately’ for Case 1 and “Not Performed 
Adequately’ for Case 2. 

1. For Case 1, the project plan was developed, and passed the 
peer review and senior management approval processes. 

2. For Case 2, the plan was developed, and underwent the 
peer review process, but not until well into the performance 
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period. It was not approved by senior management until 
after the project work was done. 

3. The team lead (senior management) was involved in several 
informal reviews and the peer review. However, the plan 
was not approved until after the project work was done. 

• Generic Goal (GG) 2 - “The process is institutionalized as a managed 

process.” 

o Generic Practice (GP) 2.1 - “Establish and maintain an 

organizational policy for planning and performing the Project 
Planning process” was “Performed Adequately’ for both Case 1 
and Case 2. 

1. The organization has a project planning policy albeit very 
general. The process did not change from the time the Case 
1 project was executed to the time the Case 2 project was 
executed. 

o GP2.2 - “Establish and maintain the plan for performing the Project 
Planning process” was “Performed Adequately’ for Case 1 and 
“Not Performed Adequately’ for Case 2. 

1. The organization project plan procedures describe how to 
create a project plan. 

2. Schedules for both Case 1 and Case 2 projects have 
development milestones for the plans, under a main “Project 
Planning” category. 

3. For Case 1, the project plan was developed, and passed the 
peer review and senior management approval processes. 

4. For Case 2, the plan was developed but did not go through 
the same review and approval processes. 

o GP2.3 - “Provide adequate resources for performing the Project 
Planning process, developing the work products, and providing the 
services of the process” was “Performed Adequately’ for Case 1 
and “Not Performed Adequately’ for Case 2. 


44 




1. Resources were provided for both projects to do planning, as 
evidenced by the project plans developed. 

2. However, for Case 2, primary responsibility was passed from 
one to another person more than once. When action wasn’t 
taken, no “quick” attempt was made by management to 
reassign this responsibility. 

o GP2.4 - “Assign responsibility and authority for performing the 
process, developing the work products, and providing the services 
of the Project Planning process” was “Performed Adequately' for 
Case 1 and “Not Performed Adequately’ for Case 2. 

1. Resources were provided for both projects to do planning, as 
evidenced by the project plans developed. 

2. However, for Case 2, primary responsibility was passed from 
one to another person more than once. When action wasn’t 
taken, no “quick” attempt was made by management to 
reassign this responsibility. 

o GP2.5 - “Train the people performing or supporting the Project 
Planning process as needed” was “Performed Adequately’ for 
both Case 1 and Case 2. 

1. For both projects, training consisted of previous experience 
of the members of a small management team; in fact, the 
same team was used for both projects. 

2. In addition, on the job training was provided to some team 
members by functional leads, and some had a college 
course in project management, which included planning 
aspects. 

o GP2.6 - “Place designated work products of the Project Planning 
process under appropriate levels of configuration” was “Not 
Performed Adequately' for Case 1 or Case 2. 
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1. All project plans, schedules, and other project planning 
artifacts were stored in the team shared folder. However, 
the documents were not put into ClearCase, the designated 
CM tool. 

2. The Case 2 project plan was put into ClearCase, but not until 
after the project was over. 

o GP 2.7 - “Identify and involve the relevant stakeholders of the 
Project Planning process as planned” was “Performed 
Adequately’ for both Case 1 and Case 2. 

1. Relevant stakeholders were all part of the team that 
developed and reviewed the project plans for both projects. 

2. These activities were monitored by the QA group, during 
project and senior management review meetings. 

o GP2.8 - “Monitor and control the Project Planning process against 
the plan for performing the process and take appropriate corrective 
action” was “Performed Adequately’ for both Case 1 and Case 2. 

1. Project plans for both Case 1 and Case 2 underwent the 
peer review process. However, corrective action was not 
taken to address getting plan changes incorporated. 

2. These activities were monitored by the QA group, during 
project and senior management review meetings. 

o GP2.9 - “Objectively evaluate adherence of the Project Planning 
process against its process description, standards, and procedures, 
and address noncompliance” was “Performed Adequately’ for 
both Case 1 and Case 2. 

1. Planning activities were monitored (albeit informally) by the 
QA group, during project and senior management review 
meetings. 

o GP2.10 - “Review the activities, status, and results of the Project 
Planning process with higher-level management and resolve 
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issues” was “Performed Adequately’ for Case 1 and “Not 
Performed Adequately' for Case 2. 

1. Senior management review meetings were conducted for the 
Case 1 project on a regular basis throughout the entire 
project. 

2. Only 2 senior management review meetings were conducted 
early on for the Case 2 project. 

Generic Goal (GG) 3 - “The process is institutionalized a defined 

process.” 

o GP 3.1 - “Establish and maintain the description of a defined 
Project Planning process” was “Performed Adequately’ for both 
Case 1 and Case 2. 

1. Four organization level procedures were identified; the 
“Software Schedule Procedure”, the “Senior Management 
Review Procedure”, the “Software Development Plan 
Procedure”, and the “Software Project Planning and 
Estimating Procedure.” 

o GP 3.2 - “Collect work products, measures, measurement results, 
and improvement information derived from planning and performing 
the Project Planning process to support the future use and 
improvement of the organization’s processes and process assets” 
was “Not Performed Adequately’ for Case 1 or Case 2. 

1. Metrics on the planning process (such as the number of 
pages for the project plan, length of time to prepare the plan, 
and number of peer reviews held for previous projects) was 
available. However, the data was not used to generate 
estimates for either Case 1 or Case 2. 

2. Lessons learned from previous projects and for both of these 
projects were identified, collected, and analyzed to assist in 
planning the next project. 




3. All of these activities were done on an informal basis. Data 
was not complied and analyzed with process improvement in 
mind. 

B. CASE 1 ANALYSIS 

1. Evaluation Results 

The following observations were made based on ratings assigned: 

• 100% practices were performed to some degree (all were rated either 
“Not Performed Adequately’, or “ Performed Adequately’ - none 
were rated “Not Performed).’’ All practices are being performed to 
some extent. 

• Overall 22/26 or 85% practices were rated “Performed Adequately'. 
This describes the success of most practices associated with each of 
the three goals in this Process Area indicating that the improvements 
will not need to be concentrated on one particular goal. 

2. Lessons Learned Reported 

a. The Project Lacked a Single Unified Software Project 
Management Plan 

The lack of a single coordinated plan resulted in individual 
functional plans whose schedules were not necessarily in synch. This led to 
significant delays in getting senior management approval. 

b. The Lab Facility Had Much Unplanned Lab Down Time 

Organization lab facility schedules and resources directly affected 
the development and test efforts, in fact, forced the need to utilize resources at 
the industry partner facility at their convenience (with some inconvenience to the 
developers and testers). As a result there were significant delays in development 
and testing activities that required lab use. 


48 



C. CASE 2 ANALYSIS 

1. Evaluation Results 

• 100% practices were performed to some degree (all were rated either 
“Not Performed Adequately’, or “ Performed Adequately’ - none 
were rated “Not Performed ). All practices were performed to some 
extent. 

• Overall, 16/26 or 62% practices were rated “Performed Adequately’. 
This describes the success of most practices associated with each of 
the three goals in this Process Area indicating that the improvements 
will not need to be concentrated on one particular goal. 

2. Lessons Learned Reported 

a. The Project Plan Took too long to Produce and Review 

The project plan was not completed, peer reviewed, and approved 
by senior management until well after the work was done. New persons, the 
industry partner, and even existing team members did not know about and 
understand all parts of the plan so there was confusion about several aspects, 
especially how one group handed off their work to the next group (development, 
test, SQA, CM). 

b. Effort Estimates Were Not Revised to Reflect Increased 
Scope 

Early on in the project there was a significant increase in the scope. 
Effort estimates were not revised, peer reviewed, and approved by senior 
management. As a result, after that point it was impossible to compare actual to 
estimated data (it did not make sense anymore. 

c. Effort Data Was Not Organized Consistently 

There does not appear to be a consistent way to organize the data 
(i.e., to record time spent on multiple projects, or for multiple tasks on the same 
project). SQA records ALL data for 100% hours spent in one worksheet for each 
group member. This enables overhead/direct effort and project-to-project 
comparisons. There appears to be a separate worksheet for some activities. 

Yet, the data it contains has not been considered in audits. In some cases there 
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is no way to determine which project the data is recorded for, and the existing 
data does not appear to be complete. 

D. COMPARISON OF CASE 1 TO CASE 2 

1. Goals 

• Specific Goal (SG) 1 - Estimates of project planning parameters are 
established and maintained. 

• Specific Goal (SG) 2 - A project plan is established and maintained as 
the basis for managing the project. 

• Specific Goal (SG) 3 - Commitments to the project plan are established 
and maintained. 

• Generic Goal (GG) 2 - The process is institutionalized as a managed 
process. 

• Generic Goal (GG) 3 - The process is institutionalized a defined 
process. 

Case 1 - SG 1 -4/4 or 100% 

SG 2 - 5/7 or 71 % 

SG 3-3/3 or 100% 

GG 2 - 8/10 or 80% 

GG 3 - Vz or 50% 

Case 2- SG 1 -4/4 or 100% 

SG 2 - 4/7 or 57% 

SG 3 - 2/3 or 67% 

GG 2-4/10 or 40% 

GG 3- 1/2 or 50% 

Unfortunately, no overall goal improvements can be demonstrated by this 
data, only digression for SG 3 and GG 2. 

2. Practices 


Practice 

Case 1 

Case 2 

Not 

Performed 

Adequately 

Performed 

Adequatel 

y 

Not 

Performed 

Adequately 

Performed 

Adequately 

SP 1.1 


X 


X 

SP 1.2 


X 


X 

SP 1.3 


X 


X 

SP 1.4 


X 


X 

SP 2.1 

X 


X 
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SP 2.2 


X 

X 


SP 2.3 


X 


X 

SP 2.4 


X 


X 

SP 2.5 


X 


X 

SP 2.6 


X 


X 

SP 2.7 

X 


X 


SP 3.1 


X 


X 

SP 3.2 


X 


X 

SP 3.3 


X 

X 


GP 2.1 


X 


X 

GP 2.2 


X 

X 


GP 2.3 


X 

X 


GP 2.4 


X 

X 


GP 2.5 


X 


X 

GP 2.6 

X 


X 


GP 2.7 


X 


X 

GP 2.8 


X 


X 

GP 2.9 


X 


X 

GP 2.10 


X 

X 


GP 3.1 


X 


X 

GP 3.2 

X 


X 



Table 2 Summary of Project Planning Evaluation Results 


The results do not demonstrate improvement from Case 1 to Case 2 in 
any project planning practice evaluated. They do show digression in the 
following practices (highlighted rows); 

• SP 2.2 - Identify and analyze project risks. 

• SP3.3 - Obtain commitment from relevant stakeholders responsible for 
performing and supporting plan execution. 

• GP2.2 - Establish and maintain the plan for performing the Project 
Planning process. 

• GP2.3 - Provide adequate resources for performing the Project 
Planning process, developing the work products, and providing the 
services of the process. 
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• GP2.4 - Assign responsibility and authority for performing the process, 
developing the work products, and providing the services of the Project 
Planning process. 

• GP2.10 - Review the activities, status, and results of the Requirements 
Management process with higher-level management and resolve 
issues. 

In addition, results indicate that for both projects, the following practices 
were “Not Performed Adequately"’ 

• SP 2.1 - Establish and maintain the project’s budget and schedule. 

• SP2.7 - Establish and maintain the overall project plan content. 

• GP2.6 - Place designated work products of the Project Planning 
process under appropriate levels of configuration. 

• GP 3.2 - Collect work products, measures, measurement results, and 
improvement information derived from planning and performing the 
Project Planning process to support the future use and improvement of 
the organization’s processes and process assets. 

3. Comparison of Evaluation Results to Lessons Learned 

Examination of the comments from the evaluation results for which both 
projects had “Not Performed Adequately” rating show that while for both Case 1 
and Case 2, project plans were established, they were not formally maintained or 
controlled for either project (to reflect changes in scope, schedule, or effort 
estimates). Some metrics were collected, some artifacts were stored, and some 
lessons learned were collected. However, those activities were also conducted 
informally, without process improvement necessarily in mind. 

The Case 1 project team noted negative impacts to the project due the 
lack of a single unified software project management plan, and the fact that the 
lab facility had much unplanned down time (lessons learned). As a result, there 
were schedule and resource misunderstandings, including delays in development 
and testing. These results are a logical conclusion (could have been predicted) 
given the process evaluation results. 
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The Case 2 project had additional practices that received the “Not 
Performed Adequately” rating (noted above). The Case 2 project digressed 
quickly by never establishing a requirements baseline (which for this organization 
translates into following the management approval process), by not following 
through on project and senior management review meetings, and more important 
by not having a dedicated project engineer (manager). The project team and 
senior management knew what to do (they had been done well for the Case 1 
project). However, they lacked the discipline required to follow through with 
previously followed processes. Without conducting a formal project planning 
process audit, the project management activities degraded slowly, without much 
notice. After all, the team worked to the draft project plans and products were 
getting generated in spite of the circumstances. However, when project 
management duties fell by the wayside after obvious need several team 
members became frustrated 

The Case 2 project team noted that the Project Plan took too long to 
produce and review, effort estimates were not revised to reflect increased scope, 
and effort data was not organized consistently. As a result, there were 
misunderstandings, miscommunications, and a general lack of commitment and 
interest in completing the project plan, and recording effort data. The decisions 
(not) made put the team in the position whereby it became necessary to rely on 
the small size and experience, instead of disciplined project management 
activities. The lack of team and senior management interaction caused the team 
to drift apart in the way effort data was organized and recorded. 

Again, it is interesting to note that one characteristic of a CMMI Level 1 
organization is that people create their own processes to do the work, creating 
the potential for duplication, and even conflict later on in the project. This is 
exactly what happened with the project plan and the effort data records. 

While the organization did not have outstanding performance on all 
practices, it was demonstrated that something was being done by the project 

team in every process area. The transition from simply managing requirements 
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to actually managing plans takes place as the organization advances from Level 
2 to Level 3. It appears that the source of risk lies within managing the plans for 
this organization. 

This information can be used to identify a business goal on which to base 
targeted process improvement, such as “to improve the project plan baseline and 
change process.” 

E. SUGGESTED IMPROVEMENTS 

As a result of the above Case 1 and Case 2 analysis, the following list of 
improvement suggestions was compiled. Some suggestions are a direct 
reflection of the lessons learned reported, some are specific to CMMI practice 
requirements that need to be fulfilled, and some are more general and would 
contribute indirectly to all reported deficiencies. The list can be used to improve 
the project performance by targeting process improvement in these areas. 

• Make sure a dedicated resource is assigned to perform the project 
engineer (manager) role (to follow up on project plan peer review and 
approval process, project and senior management review meetings, 
the change control process, etc.). If a dedicated resource is 
unavailable, duties should be reassigned aross available resources (in 
this case, Functional Leads). 

• A single Software Project Management Plan (SPMP) should be 
developed that coordinates all of the functional area plans for the 
project - Software Development Plan (SDP), Risk Management Plan 
(RMP), Software Test Plan (STP), Software Configuration 
Management Plan (SPMP), Software Quality Assurance Plan (SQAP), 
and Laboratory Management Plan (LMP). 

o The schedule and resources available for the organization lab 
facilities should be included in planning and estimating activities. 
The acquisition of laboratory equipment and any related down time 
for installation should be included in overall project planning. This 
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would ensure that lower priority items would be managed at the 
same time immediate problems are being resolved quickly. This 
would prevent significant delays in development and testing 
activities that require lab use. The lab played a huge part in the 
ability to follow the planned processes, for both cases. This implies 
that management needs to improve the lab situation (i.e., provide 
more staffing/resources/oversight). 

o A new focused process for producing the plan is needed. The plan 
can be stripped down to its essential elements to further reduce the 
time it takes to produce and review. Finish the plan well before 
delivering software. Define specific authority and criteria to catch a 
significant slip in plan completion. 

• Make sure project plans (including schedule and scope documents) 
are updated to reflect program level changes. Use “baselining” 
(accomplished at the organization by completing the senior 
management approval process) to keep a record of current 
agreements made for the duration of the project. 

• Updates to effort estimates must be documented and maintained. 
Correct information on the location of the data and who should be on 
the audit roster needs to be communicated to the auditor. 

• Develop a plan for effort data organization such that short term (ease 
of entry) and long term (combining data for overall comparison 
purposes) goals are considered. 

• Develop questions and metrics to measure accomplishments in 
response to the business goal to “improve the project planning 
process.” 

• Archive all project planning work products using the organization’s 
designated CM tool. 
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• Conduct a formal audit on the project planning process at 
during project execution. 

• Update policy, process, and procedure documentation 
additional activities. 


least once 


to reflect 
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VI. PROJECT MONITORING AND CONTROL STUDY 


The results and analysis of the project monitoring and control process for 
the two case studied are shown in the following subsections. An outline of the 
goals and associated practices defined for the CMMI Project Monitoring and 
Control Process Area, a score for each project studied (Case 1 and Case 2), and 
justification for each rating is first provided. A general analysis of the evaluation 
results for each case follows the data presentation. The analysis includes 
presentation and comparison to the post-mortem lessons learned to determine if 
there are correlations with the evaluation results. Note that the lessons are 
unique to the individual projects, and that only the lessons that apply to the area 
of study are included. Finally, a comparison of the two cases is made and a list 
of suggested improvements is provided. 

A. CASE 1 AND CASE 2 EVALUATION RESULTS 

The goals and associated practices defined for the CMMI Project 
Monitoring and Control Process Area, and individual practice ratings assigned by 
the project team are as follows: 

• Specific Goal (SG) 1 - “Actual performance and progress of the project 
are monitored against the project plan.” 

o Specific Practice (SP) 1.1 - “Monitor (after documenting initial 
estimates, changed estimates if project parameters changed) 
actual values of the project planning parameters against the project 
plan” was “Performed Adequately’ for both Case 1 and Case 2. 
Note: “Attributes” are “inputs to estimating models, such as 
complexity, or skills/effort, or previous estimates of the software 
changes” (i.e., values that affect calculation of the size of a project). 

1. The Case 1 project schedule, effort, and scope were 
identified, reviewed, and baselined (approved by senior 
management) during planning stages of the project. 
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Subsequent review and revision of this data took place at 
regular project and senior management review meetings. 

2. The Case 2 project schedule, effort, and scope were 
identified and reviewed during planning stages of the project. 
However, they were never baselined (approved by senior 
management). Subsequent review and revision of this data 
took place at regular project and senior management review 
meetings. However, this activity dropped off early in the 
project as the leadership in the project engineer (manager) 
role also dropped off. The project team was able to make 
planned scope deliveries within schedule, cost anyway. 

3. For both projects, project plans were based on parameters. 
Resources and delivery contents are allocated, and affected 
by, actual attributes (i.e., problem analysis may reveal a 
lesser or greater actual complexity factor). 

o SP1.2 - “Monitor commitments against those identified in the 
project plan” was “Performed Adequately ’ for both Case 1 and 
Case 2. 

1. Case 1 project commitments (internal and external) to 
scope, schedule, and cost were reviewed regularly at 
internal project and senior review management review 
meetings, and external overarching integrated product team 
meetings. This demonstration is evident from copies of 
associated minutes and presentation materials. 

2. Case 2 external commitments were reviewed with industry 
partners on a regular basis. Internal commitments were 
reviewed initially at project and senior management review 
meetings. However, this activity dropped off early in the 
project as the leadership in the project engineer (manager) 
role also dropped off. The project team was able to make 
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planned scope deliveries within schedule and cost anyway 
since commitments were reviewed with the software and test 
teams, even though it was a less formal process. 

3. Also, planned SQA audits were conducted for both projects, 
in addition to monitoring peer review activity, 
o SP1.3 - “Monitor risks against those identified in the project plan” 
was “Performed Adequately’ for Case 1 and “Not Performed 
Adequately' for Case 2. 

1. Case 1 risks were identified, reviewed, and baselined 
(approved by senior management) during planning stages of 
the project. A separate Risk Management Plan was 
prepared. Subsequent review and revision of this data took 
place at regular project and senior management review 
meetings. 

2. Case 2 risks were identified and reviewed during planning 
stages of the project, as a Risk Management section was 
included in the project plan. Subsequent review and revision 
of this data took place at regular project and senior 
management review meetings. However, this activity 
dropped off early in the project as the leadership in the 
project engineer (manager) role also dropped off. The 
project team was able to make planned scope deliveries 
within schedule and cost anyway. However, the project 
team indicated that monitoring risks informally was not 
sufficient. 

o SP1.4 - “Monitor the management of (internal) project data 
(schedule, cost in terms of effort, and scope) against the project 
plan” was “Performed Adequately’ for both Case 1 and Case 2. 
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1. Project data was interpreted as “documents required to 
support the project such as work products, action item lists, 
meeting minutes, or lessons learned reports.” 

2. For both projects, the plan to manage project data was 
provided in the project plan. It included directions for 
generation and distribution of both deliverable and non¬ 
deliverable work products, CM (SCR database), and QA 
artifacts. Meeting minutes, QA audit results, and the project 
action item list served as records used to monitor the 
management of project data. 

3. The Case 1 project schedule, effort, and scope were 

identified, reviewed, and baselined (approved by senior 
management) during planning stages of the project. 

Subsequent review and revision of this data took place at 

regular project and senior management review meetings. 

4. The Case 2 project schedule, effort, and scope were 

identified and reviewed during planning stages of the project. 
However, they were never baselined (approved by senior 
management). Subsequent review and revision of this data 
took place at regular project and senior management review 
meetings. However, this activity dropped off early in the 
project as the leadership in the project engineer (manager) 
role also dropped off. The project team was able to make 
planned scope deliveries within schedule and cost anyway. 

o SP1.5 - “Monitor stakeholder involvement against the project plan” 
was “Performed Adequately' for both Case 1 and Case 2. 

1. Case 1 stakeholder involvement was identified, reviewed, 
and baselined (approved by senior management) during 
planning stages of the project. Subsequent review and 
revision of this data took place at regular project and senior 
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management review meetings, overarching integrated 
product team or software meetings with the industry partner. 

2. Case 2 stakeholder involvement was identified and reviewed 
during planning stages of the project. However, they were 
never baselined (approved by senior management). 
Subsequent review and revision of this data took place at 
regular project and senior management review meetings, 
overarching integrated product team or software meetings 
with the industry partner. However, internal senior 
management review and project meeting activities dropped 
off early in the project as the leadership in the project 
engineer (manager) role also dropped off. The project team 
was able to make planned scope deliveries within schedule 
and cost anyway. External stakeholders were involved, and 
when there was a schedule misunderstanding, it was an 
exception (it was not typical for the industry partner not to 
have informed the organization of a critical change), 
o SP1.6 - “Periodically review the project’s progress, performance, 
and issues” was “Performed Adequately’ for both Case 1 and 
Case 2. 

1. Case 1 project progress, performance, and issues were 
identified (in terms of software drops), reviewed, and 
baselined (approved by senior management) during planning 
stages of the project. Subsequent review and revision of this 
data took place at regular project and senior management 
review meetings. 

2. Case 2 project progress, performance, and issues were 
identified (in terms of software drops) and reviewed during 
planning stages of the project but never baselined (approved 
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by senior management). Subsequent review and revision of 
this data took place at regular project and senior 
management review meetings but this activity dropped off 
early in the project as the leadership in the project engineer 
(manager) role also dropped off. The project team was able 
to make planned scope deliveries within schedule, cost 
anyway. The project team began to use meetings with 
industry partner and internal “change control board” 
meetings as a forum to exchange project information. While 
the process was less formal, it was enough, 
o SP1.7 - “Review the accomplishments and results of the project at 
selected project milestones” was “Performed Adequately' for both 
Case 1 and Case 2. 

1. “Milestones” was interpreted as “software releases” for the 
organization. 

2. Case 1 project progress, performance, and issues were 
identified (in terms of software drops), reviewed, and 
baselined (approved by senior management) during planning 
stages of the project. Subsequent review and revision of this 
data took place at regular project and senior management 
review meetings. In addition, planned product audits were 
conducted following the delivery of each software drop 
(including QA build/functional/configuration/test tool, and 
change control board audits). In addition, the software and 
test leads review product release notes to ensure that the 
results of change control board activities were reflected. 

3. Case 2 project progress, performance, and issues were 
identified (in terms of software drops) and reviewed during 
planning stages of the project but never baselined (approved 
by senior management). Subsequent review and revision of 
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this data took place at regular project and senior 
management review meetings. However, this activity 
dropped off early in the project as the leadership in the 
project engineer (manager) role also dropped off. The 
project team was able to make planned scope deliveries 
within schedule, cost anyway. The project team began to 
use meetings with the industry partner and internal “change 
control board” meetings as a forum to exchange project 
information. While the process was less formal, it was 
enough. 

• Specific Goal (SG) 2 - “Corrective actions are managed to closure 
when the project’s performance or results deviate significantly from the 
plan." 

o SP2.1 - “Correct and analyze the issues and determine the 
corrective actions necessary to address the issues” was 
“Performed Adequately’ for both Case 1 and Case 2. 

1. Note that the goal (SG 2) refers to the project plan versus 
actual performance. 

2. Issues were identified, assigned (resource and expected 

closure date), and logged on the Case 1 project action item 
list. Team members would use the project or senior 

management review meetings to make collective decisions 
on how to resolve issues. Subsequent review and revision 
of this data took place at regular project and senior 
management review meetings 

3. Issues were identified, assigned (resource and expected 

closure date), and logged on the Case 2 project action item 
list. Team members would use the project or senior 

management review meetings to make collective decisions 
on how to resolve issues. 
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4. Subsequent review and revision of this data took place at 
regular project and senior management review meetings. 
However, this activity dropped off early in the project as the 
leadership in the project engineer (manager) role also 
dropped off. The process became less formal, and the 
project team was still able to make planned scope deliveries 
within schedule, cost anyway. Meetings with the industry 
partner and internal change control board meetings were 
used as forums to exchange project information, 
o SP2.2 - “Take corrective action on identified issues” was 
“Performed Adequately’ for both Case 1 and Case 2. 

1. Because of the activities conducted (identified above), 
issues got resolved. Both projects had action item lists to 
demonstrate that corrective action was taken, 
o SP2.3 - “Manage corrective actions to closure” was “Performed 
Adequately’ for both Case 1 and Case 2. 

1. Because of the activities conducted (identified above), 
issues got resolved. Both projects had action item list’s to 
demonstrate that corrective action was taken, and closure 
reached. 

• Generic Goal (GG) 2 - “The process is institutionalized as a managed 
process.” 

o Generic Practice (GP) 2.1 - “Establish and maintain an 

organizational policy for planning and performing the Project 
Monitoring and Control” process was “Performed Adequately’ for 
both Case 1 and Case 2. 

1. The organization has an established “Software Project 
Tracking and Oversight Policy.” 
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o GP2.2 - “Establish and maintain the plan for performing the Project 
Monitoring and Control process” was “Performed Adequately’ for 
Case 1 and “Not Performed Adequately’ for Case 2. 

1. “Establish and maintain” implies “baselining”, which for the 
organization translates into senior management approval 
(sign-off). 

2. A plan for monitoring and controlling the Case 1 project 
schedule, effort, and scope were established in the project 
plan, which was reviewed and baselined (approved by senior 
management) early in the project. The plan did not change. 

3. A plan for monitoring and controlling the Case 2 project 
schedule, effort, and scope were established in the project 
plan, which was reviewed early in the project but never 
baselined (approved by senior management). The plan did 
not change. 

o GP2.3 - “Provide adequate resources for performing the Project 
Monitoring and Control process, developing the work products, and 
providing the services of the process” was “Performed 
Adequately’ for Case 1 and “Not Performed Adequately’ for 
Case 2. 

1. “Resources” could be persons (such as the project manager) 
or tools (such as the effort worksheets, action item lists, or 
MS project) 

2. Senior management provided the Case 1 project 
management resource to monitor and report on the project’s 
progress, performance, and issues related to the schedule, 
effort, and scope. 

3. Senior management provided the Case 2 project 
management resource to monitor and report on the project’s 
progress, performance, and issues related to the schedule, 
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effort, and scope. Other priorities took precedence and it 
became necessary to identify a new person to fulfill this role. 
Delays in this decision, and other priorities, caused 
significant lack in these activities. 

o GP2.4 - “Assign responsibility and authority for performing the 
process, developing the work products, and providing the services 
of the Project Monitoring and Control process” was “Performed 
Adequately’ for Case 1 and “Not Performed Adequately' for 
Case 2. 

1. Senior management provided the Case 1 project 

management resource to monitor and report on the project’s 

progress, performance, and issues related to the schedule, 
effort, and scope. 

2. Senior management provided the Case 2 project 

management resource to monitor and report on the project’s 

progress, performance, and issues related to the schedule, 
effort, and scope. Other priorities took precedence and it 
became necessary to identify a new person to fulfill this role. 
Delays in this decision, and other priorities, caused 
significant lack in these activities. 

o GP2.5 - “Train the people performing or supporting the Project 
Monitoring and Control process” as needed was “Performed 
Adequately’ for both Case 1 and Case 2. 

1. Training consisted of previous experience, individual 
education, and on the job training for both the Case 1 and 
Case 2 project teams. The project team indicated that the 
training was adequate to accomplish Project Monitoring and 
Control goals. 
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o GP2.6 - “Place designated work products of the Project Monitoring 
and Control process under appropriate levels of configuration” was 
“Not Performed Adequately’ for Case 1 or Case 2. 

1. The project monitoring and control work products were 
identified as configuration items in both Case 1 and Case 2 
project plans, and were placed under CM in accordance with 
the draft. While the SPMP was not approved, the work was 
done. The NPA rating was given instead of the PA rating 
since the products are stored in the NextGen shared folders 
(not in ClearCase - our designated CM tool), 
o GP 2.7 - “Identify and involve the relevant stakeholders of the 
Project Monitoring and Control process” as planned was 
“Performed Adequately’ for Case 1 and “Not Performed 
Adequately’ for Case 2. 

1. Case 1 project progress, performance, and issues were 
identified, reviewed, and baselined (approved by senior 
management) during planning stages of the project. 
Subsequent review and revision of this data took place at 
regular project and senior management review meetings. 
External stakeholders were identified and regularly involved 
as planned. 

2. Case 2 project progress, performance, and issues were 
identified and reviewed during planning stages of the project 
but never baselined (approved by senior management). 
Subsequent review and revision of this data took place at 
regular project and senior management review meetings. 
However, this activity dropped off early in the project as the 
leadership in the project engineer (manager) role also 
dropped off. The project team was able to make planned 
scope deliveries within schedule, cost anyway. External 
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stakeholders were identified and regularly involved as 
planned. 

o GP2.8 - “Monitor and control the Project Monitoring and Control 
process against the plan for performing the process and take 
appropriate corrective action” was “Not Performed Adequately’ 
for both Case 1 and Case 2. 

1. The process was defined in the project plans for both 
projects, but reports of the data did not materialize as often 
planned, nor were they as complete as planned. No 
changes in the process or corrective action resulted from this 
omission. 

o GP2.9 - “Objectively evaluate adherence of the Project Monitoring 
and Control process against its process description, standards, and 
procedures, and address noncompliance” was “Not Performed 
Adequately’ for both Case 1 and Case 2. 

1. The organization’s overarching process for SQA discusses 
independently evaluating the project monitoring and control 
process. This was done by SQA at senior management 
review meetings (more often for Case 1 than for Case 2). 
However, since an independent formal audit was not 
performed for either project, the lesser rating was assigned. 

o GP2.10 - “Review the activities, status, and results of the Project 
Monitoring and Control process with higher-level management and 
resolve issues” was “Performed Adequately’ for Case 1 and “Not 
Performed Adequately' for Case 2. 

1. Case 1 project progress, performance, and issues were 
identified, reviewed, and baselined (approved by senior 
management) during planning stages of the project. 
Subsequent review and revision of this data took place at 
regular project and senior management review meetings. 
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External stakeholders were identified and regularly involved 
as planned. 

2. Case 2 project progress, performance, and issues were 
identified and reviewed during planning stages of the project 
but never baselined (approved by senior management). 
Subsequent review and revision of this data took place at 
regular project and senior management review meetings. 
However, this activity dropped off early in the project as the 
leadership in the project engineer (manager) role also 
dropped off. The project team was able to make planned 
scope deliveries within schedule, cost anyway. External 
stakeholders were identified and regularly involved as 
planned. 

• Generic Goal (GG) 3 - “The process is institutionalized a defined 
process.” 

o GP 3.1 - “Establish and maintain the description of a defined Project 
Monitoring and Control process” was “Performed Adequately” for 
both Case 1 and Case 2. 

1. The following organization procedures have been established, 
and have not changed during the time that both Case 1 and Case 
2 projects were executed: “Effort Tracking Procedure”, “Formal 
Reviews at Milestones Procedure”, “Development Hours Tracking 
Form”, “Verification Hours Tracking Form”, and “Weekly One- 
Liners Report Form” 

o GP 3.2 - “Collect work products, measures, measurement results, 
and improvement information derived from planning and performing 
the Project Monitoring and Control process to support the future use 
and improvement of the organization’s processes and process assets” 
was “Not Performed Adequately' for both Case 1 and Case 2. 

o 
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B. CASE 1 ANALYSIS 


1. Analysis of Evaluation Results 

The following observations were made based on process evaluation 
ratings given: 

• 100% of the practices were performed to some degree (all were rated 
either “Not Performed Adequately”, or “Performed Adequately” - none 
were rated “Not Performed”). All practices are being performed to 
some extent. 

• Overall, 18/22 or 82% of the practices were rated “Performed 
Adequately”. This describes the success of most practices associated 
with each of the three goals in this Process Area indicating that the 
improvements will not need to be concentrated on one particular goal, 
rather, a few practices in each area will need to be improved. 

2. Lessons Learned Reported 

a. The Initial Set of Identified Risks Was Not Effectively 
Tracked 

New risks were not effectively elicited from the full project staff. 
Some staff members were not aware of the risks being tracked. Training in the 
risk management process should be provided to ensure that true risks to project 
completion are identified, assessed, prioritized, mitigated, and that contingency 
plans are in place should a risk be realized. The current set of risks should be 
available to all project staff in a shared repository. For the Case 1 project, not all 
of the initial risks were true risks that could affect project completion. The initial 
set of identified risks was not effectively tracked. New risks were not effectively 
elicited from the full project staff. Some staff members were not aware of the 
risks being tracked. 
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b. More Complicated Problems ere Being Resolved, But the 
Effort Audits Didn’t Show How Much Effort Was Expended 
On Each Problem 

There should be more detail recorded in planning and estimating 
resource requirements. For Case 1, hours estimates were not done over time 
intervals. Instead, there was just one big lump sum. More complicated problems 
were being resolved, but the effort audits didn’t show how much effort was 
expended on each problem. With other projects competing for human resources, 
we need to keep in mind that the project can lose people. 

c. One Large All-Encompassing Schedule Became Too 
Complex and Unwieldy 

The project should be tracked at two levels - project and functional. 
All of the schedules should be posted on a shared repository so that all project 
staff are aware of key dates. For the Case 1 project, an initial attempt to track 
progress on one large all-encompassing schedule became too complex and 
unwieldy. Instead, the project was tracked by the Project Lead to a joint industry 
partner and organization schedule. The functional leads (software, test, SCM, 
and SQA) tracked the project at the software requirement level. 

d. There Was a Significant Lack of Compliance in Recording 
Effort Data 

Provide feedback to the project team on the definition, collection, 
and use of metrics. For example, the project collected the hours expended by 
each member of the project team. In order to do this effectively, each functional 
area needs well-defined categories in which to record their hours. The method 
and frequency of collecting these records needs to be fully explained to ensure 
they are up-to-date. Finally, the reason for collecting hours and how this 
information will be used also needs to be fully explained. Similar efforts are 
needed for the remaining project metrics in order to improve compliance by the 
project staff. 
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e. The Project Team Was Not Well Informed With a Summary of 
Senior Management’s Comments and Directives 

Provide feedback to the project staff on the results of the Senior 
Management Reviews (SMRs). The SMRs were effective in presenting the status 
of the project to the Associate Director (AD) and getting his directives for 
resolving management issues. However, only functional area leads attended the 
SMRs. The remaining project staff should have the option to attend. In addition, 
while the slides presented at the SMR were made available to all, SMR minutes 
were not published. This missed the opportunity to keep the project staff 
informed with a summary of senior management comments and directives. 

f. Individual Project Team Members Did Not Apply Their Own 

Experience to Help in Identifying, Assessing, Prioritizing, 
Mitigating, and Tracking Risks 

All members of the project team need training in risk management. 
Everyone needs to be involved in identifying, assessing, prioritizing, mitigating, 
and tracking risks and in preparing contingency plans in case a particular risk is 
realized. Each member should be capable of applying their own experience to 
assist the Project Lead in identifying and managing risks. This will help ensure 
that all risks that could significantly impact a project are identified. 

g. Project Risks Were Not Presented as Effectively as They 
Could Have Been 

The format for presenting the highest priority risks at Senior 
Management Reviews should effectively identify issues for senior management 
consideration. Various formats were tried. 

h. SCM Was Not Viewed as a Continuous Process Involving 
Participation by the Entire Project Staff 

SCM is not just a repository for project work products but a 
continuous process involving participation by the entire project staff. Such 
participation should lead to improvements in the process as staff members 
generate Process Change Requests (PCRs) to resolve issues. 

i. Some Metrics, Such As Effort Data, Were Difficult To Collect 

Manually and Presented Problems in the Staff Not 
Understanding the Benefit of Their Use 
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Select project metrics for the information they provide and ensure 
their ease of collection. The project staff was not always aware of what metrics 
were collected and what they would be used for. Some metrics, such as effort 
data, was difficult to collect manually. In addition, it was difficult to get all project 
team members to record data. Perhaps this was because they did not 
understand the benefits of collecting this data. 

C. CASE 2 ANALYSIS 

1. Analysis of Evaluation Results 

The following observations were made based on process 

evaluation ratings given: 

• 100% of the practices were performed to some degree (all were rated 
either “Not Performed Adequately”, or “Performed Adequately” - none 
were rated “Not Performed”). All practices are being performed to 
some extent. 

• Overall, 12/22 or 55% of the practices were rated “Performed 
Adequately”. This describes the success of most practices associated 
with each of the three goals in this Process Area indicating that the 
improvements will not need to be concentrated on one particular goal, 
rather, a few practices in each area will need to be improved. 

2. Lessons Learned Reported 

a. Schedules and Key Project Artifacts Were Not Kept Up to 
Date and Visible 

Schedules and key project artifacts were not presented at every 
meeting, nor were responsibilities for maintenance assigned. In addition, there 
were no defined standard access rules for these artifacts. 

b. Elements of the Plan Were Not Performed 

Project meetings, senior management review meetings, and risk 
management activities were not executed. Plans were not coordinated and 
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action items were not recorded or tracked to closure. The project lacked a full 
time project engineer (manager) to lead these efforts. 

c. Project Metrics Not Accurate and are Not Being Used 

Essential project data was not defined and did not have 

spreadsheets to manipulate and store the data, did not have built-in reporting 
requirements for project meetings and senior management reviews. The data 
collection process needs to be reviewed and refined for ease and efficiency. 

d. An Overall Project Schedule Was Non-Existent 

The lack of a project schedule resulted in differing milestones 
between the developers, testers, CM, SQA, and the industry partner. 

e. Lack of Project Manager 

The lack of a project lead caused much frustration due to the lack 
of coordination, both internal and external. 

f. Lack of Commitment to Effort Recording/Reporting 

There were a variety of problems with actual effort data recorded. 
There was much missing data, and estimates made well past actual work done 
were likely less accurate. In addition, QA audits sometimes (mistakenly) 
included team members who were no longer working on the project, and team 
member response did not seem to greatly improve over time. It is now 
impossible to compare final estimates to actual time spent. 

g. There Was a Lot of Missing Data and Much Data is 
Recorded a Considerable Amount of Time After Actual Time 
is Spent 

There appeared to be a lot of missing data, and much data is 
recorded a considerable amount of time after actual time is spent. Either data is 
lost or, when finally recorded, may be less accurate than it could have been if 
recorded on a more regular basis. Only 33% of the Case 2 project team has 
updated their worksheets within the last month. 
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h. It Has Been Difficult to Determine if Worksheets are Current 
(and “Zero” Time Has Been Put Towards the Project) or if 
Data Entry is Lagging 

For audit purposes, the latter was assumed. If correct information 
is not communicated to the auditor, audit results may be skewed. 

/. There is Not a Consistent Naming Convention Used to 
Identify the Effort Record Worksheets 

QA worksheets use fiscal year and team member identification in 
the file names. Test group worksheets use only name for Case 2 worksheets 
and name and project identification for Case 2 worksheets in the file names. 
Development and lab worksheets use only name identification in the file names. 
Once archived, it would be difficult to determine what project the data was 
recorded for with out opening each file. 

D. COMPARISON OF CASE 1 TO CASE 2 

1. Goals 

The goals associated with the PMC PA are as follows: 

• Specific Goal (SG) 1 - Specific Goal (SG) 1 - “Actual performance and 
progress of the project are monitored against the project plan.” 

• Specific Goal (SG) 2 - “Corrective actions are managed to closure 
when the project’s performance or results deviate significantly from the 
plan.” 

• Generic Goal (GG) 2 - “The process is institutionalized as a managed 
process.” 

• Generic Goal (GG) 3 - “The process is institutionalized a defined 
process.” 

The achievement by Case 1 and Case 2 in terms of goal success can be 
described by a comparison of the number of practices given the “Performed 
Adequately” rating to the total number of practices for that goal. The results of 
this analysis follow: 
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Case 1 - SG 1 -7/7 or 100% 

SG 2 - 3/3 or 100% 

GG 2-7/10 or 70% 

GG 3- 1/2 or 50% 

Case 2 - SG 1 - 6/7 or 86% 

SG 2 - 3/3 or 100% 

GG 2-2/10 or 20% 

GG 3- 1/2 or 50% 

No overall goal improvements can be demonstrated by this data, only 
digression for SG 1, and GG 2. 


2. Practices 


Practice 

Case 1 

Case 2 

Not 

Performed 

Adequately 

Performed 

Adequatel 

y 

Not 

Performed 

Adequately 

Performed 

Adequately 

SP 1.1 


X 


X 

SP 1.2 


X 


X 

SP 1.3 


X 

X 


SP 1.4 


X 


X 

SP 1.5 


X 


X 

SP 1.6 


X 


X 

SP 1.7 


X 


X 

SP 2.1 


X 


X 

SP 2.2 


X 


X 

SP 2.3 


X 


X 

GP 2.1 


X 


X 

GP 2.2 


X 

X 


GP 2.3 


X 

X 


GP 2.4 


X 

X 


GP 2.5 


X 


X 

GP 2.6 

X 


X 


GP 2.7 


X 

X 


GP 2.8 

X 


X 


GP 2.9 

X 


X 


GP 2.10 


X 

X 


GP 3.1 


X 


X 

GP 3.2 

X 


X 



Table 3 Summary of Project Monitoring and Control Evaluation Results 
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The results do not demonstrate improvement from Case 1 to Case 2 in 
any project monitoring and control practice evaluated. They do show digression 
in the following practices (highlighted rows): 

• SP1.3 - Monitor risks against those identified in the project plan.” 

• GP2.2 - Establish and maintain the plan for performing the Project 

Monitoring and Control process. 

• GP2.3 - Provide adequate resources for performing the Project 

Monitoring and Control process, developing the work products, and 
providing the services of the process. 

• GP2.4 - Assign responsibility and authority for performing the process, 

developing the work products, and providing the services of the Project 
Monitoring and Control process. 

• GP 2.7 - Identify and involve the relevant stakeholders of the Project 

Monitoring and Control process. 

• GP 2.10 - Review the activities, status, and results of the Project 

Monitoring and Control process with higher-level management and 
resolve issues. 

In addition, results indicate that for both projects, the following practices 
were “Not Performed Adequately:” 

• GP2.6 - Place designated work products of the Project Monitoring and 
Control process under appropriate levels of configuration 
management. 

• GP2.8 - Monitor and control the Project Monitoring and Control 
process against the plan for performing the process and take 
appropriate corrective action. 

• GP2.9 - Objectively evaluate adherence of the Project Monitoring and 
Control process against its process description, standards, and 
procedures, and address noncompliance. 
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• GP 3.2 - Collect work products, measures, measurement results, and 
improvement information derived from planning and performing the 
Requirements Management process to support the future use and 
improvement of the organization’s processes and process assets. 

3. Comparison of Evaluation Results to Lessons Learned 

Examination of the comments from the evaluation results for which both 
projects had “Not Performed Adequately” rating show that designated work 
products were not put under appropriate levels of configuration management, the 
PMC process was not monitored, and corrective action was not taken when it 
was needed. Similar to other process areas evaluated, some metrics were 
collected, some artifacts were stored, and some lessons learned were collected. 
However, those activities were also conducted informally, without process 
improvement necessarily in mind. 

The Case 1 project team noted negative impacts to the fact that the initial 
set of identified risks were not effectively tracked, individual project team 
members did not apply their own experience to help in identifying, assessing, 
prioritizing, mitigating, and tracking risks, project risks were not presented as 
effectively as they could have been. The team also recognized that more 
complicated problems were being resolved but the effort audits didn’t show how 
much effort was expended on each problem. In addition, there was a significant 
lack of compliance in recording effort data, and some metrics, such as effort 
data, was difficult to collect manually and presented problems in the staff not 
understanding the benefit of their use. In general, the project team 
acknowledged that they did not think they were well informed of senior 
management’s comments and directives, and that the large all-encompassing 
schedule became too complex and unwieldy. At first glance this list seems long 
compared to the evaluation results reported, but the fact that there was not 
enough corrective action taken to correct process execution supports the entire 
list. 
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The Case 2 project had additional practices that received the “Not 
Performed Adequately” rating (noted above). The ratings were a direct result of 
inadequate risk monitoring, not maintaining the plan for the PMC process, not 
providing adequate resources and assigning responsibility and authority for 
executing the PMC process, inadequate stakeholder involvement, and not 
reviewing issues with senior management or using them to help solve problems. 

The Case 2 lessons learned was a long list including the fact that 
schedules and key project artifacts were not kept up to date and visible, elements 
of the plan were not done, project metrics not accurate and were not being used, 
an overall project schedule was non-existent, lack of project manager, and a lack 
of commitment to effort recording/reporting (much missing data and inaccurate 
data, and inconsistent worksheet organization). These are the symptoms noticed 
by the project team as a result of confusion over the project plan, and lack of 
direction from a dedicated project engineer (manager). 

Again, it is interesting to note that one characteristic of a CMMI Level 1 
organization is that people create their own processes to do the work, creating 
the potential for duplication, and even conflict later on in the project. The 
migration in effort recording activities demonstrated this characteristic. 

While the organization did not have outstanding performance on all 
practices, it was demonstrated that something was being done by the project 
team in every area. The transition from simply managing requirements to 
actually managing plans takes place as the organization advances from Level 2 
to Level 3. It appears that the source of risk lied with maintaining a dedicated 
resource for a project engineer and keeping senior management involved to help 
enforce the discipline needed to make sure corrective action was identified and 
followed through with. 

This information can be used to identify a business goal on which to base 
targeted process improvement, such as “to improve the project monitoring and 
control process.” 
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E. SUGGESTED IMPROVEMENTS 

As a result of the above Case 1 and Case 2 analysis the following list of 
improvement suggestions was compiled. Some suggestions are a direct 
reflection of the lessons learned reported, some are specific to CMMI practice 
requirements that need to be fulfilled, and some are more general and would 
contribute indirectly to all reported deficiencies. It can be used to improve the 
project performance by targeting process improvement in these areas. 

• Make sure a dedicated resource is assigned to perform the project 
engineer (manager) role (to follow up on project plan peer review and 
approval process, project and senior management review meetings, 
risk management, the change control process, etc.). 

• Schedules, plans, scope documents, and other and key project 
artifacts need to be kept up to date and visible, presented at every 
meeting. Responsibilities for maintenance of these products must be 
clearly assigned. Use “baselining” (accomplished at the organization 
by completing the senior management approval process) to keep a 
record of current agreements made for the duration of the project. 

• Training in the risk management process should be provided to ensure 
that true risks to project completion are identified, assessed, prioritized, 
mitigated, and that contingency plans are in place should a risk be 
realized. The current set of risks should be available to all project staff 
in a shared repository. A standard format for presenting the highest 
priority risks at Senior Management Reviews should be developed. 

• There should be more detail recorded in planning and estimating 
resource requirements. The project should be tracked at two levels - 
project and functional. All of the schedules should be posted on a 
shared repository so that all project staff are aware of key dates. 

• Select project metrics for the information they provide and ensure their 
ease of collection. More feedback on definition, collection, and use of 
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metrics should be provided to project team members. The categories, 
method, frequency of collecting these records, and the reason for 
collecting hours and how this information will be used also needs to be 
well defined and fully explained. Similar efforts are needed for the 
remaining project metrics in order to improve compliance by the project 
staff. 

• Provide more feedback to the project staff on the results of the Senior 
Management Reviews (SMRs). 

• Review and renew the commitment to record and collect effort data to 
demonstrate that this is a valuable and necessary activity. Develop 
standard naming conventions for effort worksheet file names and effort 
categories, a way to indicate when the worksheet was last updated, 
and a standard way to add/delete team members from the audit 
roster. 

• Develop questions and metrics to measure accomplishments in 
response to the business goal to “improve the project monitoring and 
control process.” 

• Archive all project planning work products using the organization’s 
designated CM tool (ClearCase). SCM needs to become a continuous 
process involving participation by the entire project staff (rather than 
just a repository for work products). Staff members should generate 
Process Change Requests (PCRs) to resolve issues. 

• Conduct a formal audit on the project monitoring and control process at 
least once during project execution. 

• Update policy, process, and procedure documentation to reflect 
additional activities. 
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VII. CONCLUSIONS 


This study showed that while the organization is performing to some 
extent in every area of the project management practices evaluated, there is still 
much room for improvement. The evaluation results did not demonstrate 
improvement of any one practice, there were many practices for which both 
projects were rated “Not Performed Adequately,” and there were many cases for 
which the more recent project (Case 2) got a lower rating than the less recent 
project (Casel), i.e., evaluation results showed some digression from one project 
to the next. This conclusion was reached by consensus of a small group of 
project team members who participated in reviewing the results. It matched the 
general perception held by the team, and management. 

Given that the project team was highly successful at delivering planned 
scope, within planned schedule and fixed cost, these results are surprising since 
team members have remarked about how much more organized and consistent 
the process seems to be than ever before (albeit they would be referring to ail 
processes used, not just project management processes). Why then doesn’t 
this show up in the evaluation results? 

The project team is small (less than 20 people), and most of the same 
people fulfilled the same duties on both projects studied. The projects were 
similar (unique iterations of software maintenance efforts), and the staff were 
experienced (from previous iterations of the same type of projects). The 
advantage of the circumstances implies that the team was able to rely on 
experienced staff to do whatever needed to be done to make the delivery on 
time, in spite of process digression. 

So why should the organization even do process improvement since 
achieving a certain CMMI level cannot improve on the already great record? 
One valid reason might be to satisfy a bidding requirement to gain more 
government contracts, to in effect “keep up with industry trends” by advertising 
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the organization’s achievement. But what is the internal value for this pursuit? 
Does the organization need to improve its effectiveness and efficiency? 

One way to answer this question is to look closer at one aspect of this 
evaluation -recording, collection, analysis, and reporting of effort data. The 
project team may ask themselves if they even know how effectively and 
efficiently they perform various tasks? There were many problems related to 
these activities (determining estimates versus actual time spent on categories of 
tasks). Improving this process would help the team to gather more accurate 
data, which would enable the team to discover what the measurements mean in 
terms of productivity. Once understood quantitatively, process improvements 
could be targeted for certain processes that affect effectiveness and efficiency. 
More important, the organization will have the capability to demonstrate these 
skills to customers with data, instead of opinions. This could be used as a 
valuable marketing tool. 

Further, as processes become more consistent, individual frustrations with 
ad hoc methods will begin to disappear. When the team begins to demonstrate 
enough discipline to learn, perform, and improve the same processes, the team 
will come together to do more of the same work (instead of each person 
improving their own way of doing things, which is likely different that the way 
another person performs the same task). 

One test of process improvement is whether or not it can withstand 
changes in business and personnel. The right combination of striving for 
consistency and then accepting more change will provide a successful balance. 
The organization just needs to know where to start. 

It sometimes seems that examination and acknowledgement of the 
reasons previous process improvement efforts did not take hold is too often 
glossed over, and the information not used. Getting comfortable with exposing 
barriers could be the very key to making a change stick, or at least help to 
prevent doing the same thing, and getting the same (not as good as we want) 
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results. Even though the barriers do not usually turn out to be rocket science, it 
is often difficult to uncover and overcome them, especially in an organization 
where things have been done in the same ways, and by the same people, for 
years. Add that to the fact that making even minor changes for most of us is at 
best challenging. “Starting over" has to be a way of life to get different results. 
And there has to be some strong motivation, even for simple changes. If the 
organization is committed to preparation for expanding the ability to grow the 
SOW it must be strongly emphasized. 
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VIII. POTENTIAL BENEFITS 


The following list outlines some predicted benefits that should take place 
after implementation of the suggested process improvements discussed in this 
paper. Some benefits of this study have already been realized, as all of the work 
done by this organization in the last few years have planted seeds that are now 
presenting opportunities. Conducting this study has served as a training tool 
among the participants and stirred many very good additional questions. 

A. INTERNAL TO THE ORGANIZATION 

The primary benefits internal to the organization include: 

• Increased Awareness of the current organization processes, how they 
compare to CMMI practices, and specific ideas for valuable 
improvements 

• A better understanding of the value of collecting and using lessons 
learned 

• An improved problem solving environment (better communication 
leads to greater understanding, which translates into 
acknowledgement, then comfort) 

• Opening and strengthening communication between the project team 
and senior management will help to uncover and prevent further 
misunderstanding about current processes as they work together to 
devise the best process solutions 

• Improvement suggestions that are related to low level business goals 
(i.e., “improve the requirements management process” is easier to 
measure than to “improve productivity”) can be prioritized according to 
management needs 

• The organization will learn to plan to prevent problems, instead of just 
reacting to problems. 

• Improved basic project management skills (planning, tracking, and 
reporting) 
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• Improved mid-course decision making to help manage risks 

• Increased ability to provide more accurate estimate and progress 
information to the project team, management, industry partners, and 
customers. 

• The team will become more enthusiastic as success is realized by 
owning the culture shift from relying on previous experience to 
currently defined process discipline. 

• The data needed to do earned value progress reporting will be 
available (a method of relating the scope of work to the project 
schedule and budget). 

• Increased productivity when goals between project teams and senior 
management are realigned and decisions made to enforce 
organization level consistency in implementation, so the team would be 
able to finish the project sooner, or take on more scope. 

• CMMI achievement award of as a result of a SCAMPI or official 
licensed CMMI appraisal. 

B. EXTERNAL TO THE ORGANIZATION 

Aside from some of the same benefits noted above being noticed by 
stakeholders who interact with the organization, some additional benefits that 
may be realized include the following. 

• Demonstrated contribution to the ARMY process improvement initiative 

• A new credibility that is recognized in the industry, and can be used as 
a strong marketing tool for the organization, and for the Army 
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APPENDIX - SAMPLE PROCESS AREA EVALUATION CHECKLIST 


Evaluator: SQA Engineer 
Assessment Date: 1/24/06 - 3/10/06 

Project Audited: Case 1 and Case 2 - Evaluating both projects was 
recommended by the SEPG since it will demonstrate continuous use of 
processes and artifacts. In addition, it will expose improvements, or regressions, 
from capabilities demonstrated during the project to capabilities demonstrated 
during the Case 2 project. 

Support/Review Team: Project manager, Software Development Lead, Test 
Lead, SEPG Lead, SQA Engineer, Project Manager, and CM Lead. 


Specific Practice (SP) 

Requirements Management - SP 

[Comments regarding interpretation 



Rating 


of practices as applicable to this 

NP - Not Performed, NPA - Not 

project were inserted as needed.] 


Performed Adequately, 


PA- 

- Performed Adequately or Better 


NP 

NPA 

PA 

Basis for 





Rating 

Specific Goal (SG) 1 Requirements are managed and inconsistencies 

with project plans and work products are identified. 

SP1.1 Develop an understanding with 
the requirements providers on the 
meaning of the requirements [budget, 
schedule, and delivery constraints]. 



Case 1 

Case 2 

*** Note that 
all comments 
in this column 
were removed 
for 

presentation 
of results in 
this paper to 
maintain 
project 
anonymity. 
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SP1.2 Obtain Commitment to the 
requirements from the project 
participants. This practice refers to 
the “initial” commitment (not changes) 
to a specific set of requirements. 


Case 

2 

Case 1 


SP1.3 Manage changes to the 
requirements as they evolve during 
the project. 

This practice refers only to changes 
made to the initial baseline to a 
specific set of requirements. 


Case 

1 

Case 

2 



SP1.4 Maintain bidirectional 
traceability among the requirements 
and the project plans and the work 
products. This is interpreted as 
referring to both the initial and 
changes made to the requirements. 

For the purposes of this study, it was 
interpreted as traceability between 
the external Statement of Work 
(SOW) to the organizations internal 
SOW, and to the organizations Case 

1 and Case 2 project plans. In both 
cases, the organization’s internal 
schedule was considered part of the 
plans. 


Case 

1 

Case 

2 



SP1.5 Identify inconsistencies 
between the project plans and work 
products and the requirements. This 
is interpreted as comparison between 
the (project plans and/or the work 
products) and the requirements. 


Case 

2 

Case 1 



Generic Practice (GP) 

Requirements Management - GP Rating 


NP 

NPA 

PA 

Basis for Rating 

Note that the 

comments in this 

column were 

removed for 
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presentation of 

results in this 

paper to maintain 

project 

anonymity. 

Generic Goal (GG) 2 - The process is institutionalized as a managed process. 

GP2.1 - Establish and maintain an 
organizational policy for planning 
and performing the Requirements 
Management process. 



Case 1 

Case 2 


GP2.2 - Establish and maintain 
the plan for performing the 
Requirements Management 
process. 



Case 1 

Case 2 


GP2.3 - Provide adequate 
resources for performing the 
Requirements Management 
process, developing the work 
products, and providing the 
services of the process. 



Case 1 

Case 2 


GP2.4 - Assign responsibility and 
authority for performing the 
process, developing the work 
products, and providing the 
services of the Requirements 
Management process. 



Case 1 

Case 2 


GP2.5 - Train the people 
performing or supporting the 
Requirements Management 
process as needed. 



Case 1 

Case 2 


GP2.6 - Place designated work 
products of the Requirements 
Management process under 
appropriate levels of configuration. 


Case 

1 

Case 

2 



GP 2.7 - Identify and involve the 
relevant stakeholders of the 
Requirements Management 
process as planned. 



Case 1 

Case 2 
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GP2.8 - Monitor and control the 
Requirements Management 


Case 

Case 1 


process against the plan for 
performing the process and take 
appropriate corrective action. 


2 



GP2.9 - Objectively evaluate 
adherence of the Requirements 
Management process against its 


Case 

1 



process description, standards, 


Case 



and procedures, and address 
noncompliance. 


2 



GP2.10 - Review the activities, 
status, and results of the 


Case 

Case 1 


Requirements Management 
process with higher level 
management and resolve issues. 


2 



GG3 - The process is institutionalized a defined process. 

GP 3.1 Establish and maintain the 



Case 1 


description of a defined 
Requirements Management 
process. 



Case 2 


GP 3.2 Collect work products, 


Case 



measures, measurement results, 


1 



and improvement information 


Case 



derived from planning and 
performing the Requirements 
Management process to support 
the future use and improvement of 
the organization’s processes and 
process assets. 


2 
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